Project

General

Profile

Bug #11149

Possible opencl regression

Added by Christian Kanzian over 3 years ago. Updated about 3 years ago.

Status:
Fixed
Priority:
Critical
Assignee:
-
Category:
Darkroom
Target version:
Start date:
09/12/2016
Due date:
% Done:

100%

Estimated time:
Affected Version:
git master branch
System:
all
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Hi,

The latest git version (2.1.0+1521~g1292f9) crashes if I start to edit an image in darkroom mode. I simple need to move a slider via mouse-wheel.
If I start darktable with:

darktable --disable-opencl

it works without any problem.

Here is the backtrace I tried to get with gdb. I hope it is everthing there?

chri@chk64:~/Linux/darktable$ gdb darktable
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from darktable...done.
(gdb) r
Starting program: /opt/darktable/bin/darktable 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe5ed6700 (LWP 9247)]
[New Thread 0x7fffe56d5700 (LWP 9248)]
[New Thread 0x7fffe4ed4700 (LWP 9249)]
[New Thread 0x7fffdffff700 (LWP 9250)]
[New Thread 0x7fffdf7fe700 (LWP 9251)]
[New Thread 0x7fffdeffd700 (LWP 9252)]
[New Thread 0x7fffde7fc700 (LWP 9253)]
[New Thread 0x7fffddffb700 (LWP 9254)]
[New Thread 0x7fffdd7fa700 (LWP 9255)]
[New Thread 0x7fffdcff9700 (LWP 9256)]
[New Thread 0x7fffdc55d700 (LWP 9257)]
[New Thread 0x7fffaffff700 (LWP 9258)]
[New Thread 0x7fffa7fff700 (LWP 9259)]
[New Thread 0x7fffa77fe700 (LWP 9260)]
[Thread 0x7fffa77fe700 (LWP 9260) exited]
[New Thread 0x7fffa6df9700 (LWP 9261)]
[Thread 0x7fffa6df9700 (LWP 9261) exited]
[New Thread 0x7fffa65f8700 (LWP 9263)]
[Thread 0x7fffa65f8700 (LWP 9263) exited]
[New Thread 0x7fffa5df7700 (LWP 9264)]
[Thread 0x7fffa5df7700 (LWP 9264) exited]
[New Thread 0x7fffa55f6700 (LWP 9266)]
[Thread 0x7fffa55f6700 (LWP 9266) exited]
[New Thread 0x7fffa4df5700 (LWP 9267)]
[Thread 0x7fffa4df5700 (LWP 9267) exited]
[New Thread 0x7fff9ffff700 (LWP 9268)]
[Thread 0x7fff9ffff700 (LWP 9268) exited]

(darktable:9243): Json-CRITICAL **: json_object_get_string_member: assertion 'node != NULL' failed
[Thread 0x7fffa7fff700 (LWP 9259) exited]
[New Thread 0x7fff9ffff700 (LWP 9270)]
[Thread 0x7fff9ffff700 (LWP 9270) exited]
[New Thread 0x7fff9ffff700 (LWP 9271)]
[Thread 0x7fff9ffff700 (LWP 9271) exited]
[New Thread 0x7fff9ffff700 (LWP 9272)]
[New Thread 0x7fffa4df5700 (LWP 9273)]
[Thread 0x7fffa4df5700 (LWP 9273) exited]
[New Thread 0x7fffa55f6700 (LWP 9274)]
[Thread 0x7fffa55f6700 (LWP 9274) exited]
[New Thread 0x7fffa5df7700 (LWP 9275)]
[Thread 0x7fffa5df7700 (LWP 9275) exited]
[New Thread 0x7fffad2eb700 (LWP 9276)]
[Thread 0x7fffad2eb700 (LWP 9276) exited]
[New Thread 0x7fffacaea700 (LWP 9277)]
[New Thread 0x7fffa7fff700 (LWP 9278)]
[Thread 0x7fffacaea700 (LWP 9277) exited]
[Thread 0x7fffa7fff700 (LWP 9278) exited]
[New Thread 0x7fffa77fe700 (LWP 9279)]
[Thread 0x7fffa77fe700 (LWP 9279) exited]
[Thread 0x7fff9ffff700 (LWP 9272) exited]
[New Thread 0x7fffa77fe700 (LWP 9280)]
[New Thread 0x7fffa7fff700 (LWP 9281)]
[New Thread 0x7fffacaea700 (LWP 9282)]
[New Thread 0x7fffad2eb700 (LWP 9283)]
[New Thread 0x7fffa4eb4700 (LWP 9284)]
[New Thread 0x7fff9ffff700 (LWP 9285)]
[New Thread 0x7fff934c0700 (LWP 9286)]
[New Thread 0x7fff91c72700 (LWP 9287)]
[New Thread 0x7fff91471700 (LWP 9288)]
[New Thread 0x7fff90c70700 (LWP 9289)]
[New Thread 0x7fff9046f700 (LWP 9290)]
[New Thread 0x7fff8fc6e700 (LWP 9291)]
[New Thread 0x7fff8f46d700 (LWP 9292)]
[New Thread 0x7fff8ec6c700 (LWP 9293)]
[New Thread 0x7fff8b1dd700 (LWP 9294)]
[New Thread 0x7fff8a9dc700 (LWP 9295)]
[New Thread 0x7fff8a1db700 (LWP 9296)]
[New Thread 0x7fff899da700 (LWP 9297)]
[New Thread 0x7fff7bb0b700 (LWP 9300)]
[New Thread 0x7fff7ab9c700 (LWP 9301)]
[Thread 0x7fff899da700 (LWP 9297) exited]
[Thread 0x7fff7bb0b700 (LWP 9300) exited]
[Thread 0x7fff8a9dc700 (LWP 9295) exited]
[New Thread 0x7fff8a9dc700 (LWP 9304)]
[Thread 0x7fff8a1db700 (LWP 9296) exited]
[Thread 0x7fff8a9dc700 (LWP 9304) exited]
[New Thread 0x7fff8a9dc700 (LWP 9305)]
[Thread 0x7fffaffff700 (LWP 9258) exited]
[New Thread 0x7fffaffff700 (LWP 9306)]
[Thread 0x7fff8a9dc700 (LWP 9305) exited]
[New Thread 0x7fff8a9dc700 (LWP 9307)]
[New Thread 0x7fff8a1db700 (LWP 9308)]
[Thread 0x7fff8a9dc700 (LWP 9307) exited]
[Thread 0x7fffaffff700 (LWP 9306) exited]
[New Thread 0x7fffaffff700 (LWP 9309)]
[Thread 0x7fffaffff700 (LWP 9309) exited]
[New Thread 0x7fffaffff700 (LWP 9310)]
[New Thread 0x7fff8a9dc700 (LWP 9311)]
[New Thread 0x7fff7bb0b700 (LWP 9312)]
[Thread 0x7fffaffff700 (LWP 9310) exited]
[New Thread 0x7fffaffff700 (LWP 9313)]
[New Thread 0x7fff899da700 (LWP 9314)]
[New Thread 0x7fff79d76700 (LWP 9315)]
[New Thread 0x7fff79575700 (LWP 9316)]
[New Thread 0x7fff78d74700 (LWP 9317)]
[New Thread 0x7fff6ffff700 (LWP 9318)]
[New Thread 0x7fff6f7fe700 (LWP 9319)]
[New Thread 0x7fff6effd700 (LWP 9320)]
[New Thread 0x7fff6e7fc700 (LWP 9321)]
[New Thread 0x7fff6dffb700 (LWP 9322)]
[New Thread 0x7fff6d7fa700 (LWP 9323)]
[New Thread 0x7fff6cff9700 (LWP 9324)]
[New Thread 0x7fff67fff700 (LWP 9325)]
[New Thread 0x7fff677fe700 (LWP 9326)]
[Thread 0x7fff8a9dc700 (LWP 9311) exited]
[New Thread 0x7fff8a9dc700 (LWP 9327)]
[New Thread 0x7fff58eca700 (LWP 9328)]
[New Thread 0x7fff586c9700 (LWP 9329)]
[New Thread 0x7fff57ec8700 (LWP 9330)]
[New Thread 0x7fff576c7700 (LWP 9331)]
[New Thread 0x7fff56ec6700 (LWP 9332)]
[New Thread 0x7fff566c5700 (LWP 9333)]
[New Thread 0x7fff55ec4700 (LWP 9334)]
[New Thread 0x7fff556c3700 (LWP 9335)]
[New Thread 0x7fff54ec2700 (LWP 9336)]
[New Thread 0x7fff546c1700 (LWP 9337)]
[New Thread 0x7fff53ec0700 (LWP 9338)]
[New Thread 0x7fff536bf700 (LWP 9339)]
[New Thread 0x7fff52ebe700 (LWP 9340)]
[Thread 0x7fff8a1db700 (LWP 9308) exited]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffdc55d700 (LWP 9257)]
__memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1549
1549    ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0  __memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1549
#1  0x00007fffb8921ca8 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#2  0x00007fffb8955273 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#3  0x00007fffb898d353 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#4  0x00007fffb893029d in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#5  0x00007fffb893063a in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#6  0x00007fffb88d0fcf in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#7  0x00007fffb893ec7c in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#8  0x00007ffff3b600a4 in start_thread (arg=0x7fffdc55d700) at pthread_create.c:309
#9  0x00007ffff03a587d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) bt
#0  __memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1549
#1  0x00007fffb8921ca8 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#2  0x00007fffb8955273 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#3  0x00007fffb898d353 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#4  0x00007fffb893029d in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#5  0x00007fffb893063a in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#6  0x00007fffb88d0fcf in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#7  0x00007fffb893ec7c in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#8  0x00007ffff3b600a4 in start_thread (arg=0x7fffdc55d700) at pthread_create.c:309
#9  0x00007ffff03a587d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) bt
#0  __memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1549
#1  0x00007fffb8921ca8 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#2  0x00007fffb8955273 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#3  0x00007fffb898d353 in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#4  0x00007fffb893029d in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#5  0x00007fffb893063a in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#6  0x00007fffb88d0fcf in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#7  0x00007fffb893ec7c in ?? () from /usr/lib/x86_64-linux-gnu/libamdocl64.so
#8  0x00007ffff3b600a4 in start_thread (arg=0x7fffdc55d700) at pthread_create.c:309
#9  0x00007ffff03a587d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) 

Thanks!
Christian

gdb.txt (114 KB) gdb.txt gdb output Christian Kanzian, 09/14/2016 09:27 PM
darktable_bt_IK0RNY.txt (163 KB) darktable_bt_IK0RNY.txt Dirk Sagwitz, 09/16/2016 06:12 PM
gdb.txt (108 KB) gdb.txt Ulrich Pegelow, 09/16/2016 07:04 PM
gdb.txt (150 KB) gdb.txt Roman Lebedev, 09/17/2016 04:27 PM
darktable_bt_JF9TNY.txt (88.9 KB) darktable_bt_JF9TNY.txt Albert Castells, 09/19/2016 08:37 AM

Related issues

Related to darktable - Bug #11152: darkroom crashes - opencl?Duplicate09/13/2016

Has duplicate darktable - Bug #11158: dt crash badly when working interactivelyDuplicate09/17/2016

Precedes darktable - Bug #11168: Recent pipe changes: buffer dsc caching issuesFixed

Associated revisions

Revision 05fd3dc5 (diff)
Added by Roman Lebedev about 3 years ago

dt_dev_pixelpipe_set_input(): just use get_output_format(). Fixes #11149

I bet that would not have been an issue with my MIP_F rework ^^
(PR darktable-org/darktable#1263)

History

#1 Updated by Roman Lebedev over 3 years ago

  • % Done changed from 0 to 50
  • Status changed from New to In Progress

Please install debug symbols, and start dt from console.
When it crashes, it will output a backtrace into file named /tmp/darktable_bt_XXXXXX.txt

That being said, i do not have working opencl setup so it is totally possible that i introduced an opencl-specific regression.

#2 Updated by Roman Lebedev over 3 years ago

  • % Done changed from 50 to 20
  • Status changed from In Progress to Incomplete
  • Subject changed from darktable SIGSEGV with opencl enable after commit 94f1ec7a0e4cbd155d21a364edf7341623af0284 to Possible opencl regression

#3 Updated by Christian Kanzian over 3 years ago

Roman Lebedev wrote:

Please install debug symbols, and start dt from console.
When it crashes, it will output a backtrace into file named /tmp/darktable_bt_XXXXXX.txt

[...]

Usually I do ./build.sh with default options and I get a backtrace in case of a crash. ATM there is no backtrace generated if I start darktable from a console.

chri@chk64:~/Linux/darktable$ darktable

(darktable:17607): Json-CRITICAL **: json_object_get_string_member: assertion 'node != NULL' failed
Speicherzugriffsfehler

is all I get. Thats why I did "gdb darktable" and postet the output here. How do I install "debug symbols" then to get a bt?

#4 Updated by Roman Lebedev over 3 years ago

  • Related to Bug #11152: darkroom crashes - opencl? added

#5 Updated by Kevin Gilbert over 3 years ago

I too am suffering from this bug. Would it be possible to raise the priority?

#6 Updated by Roman Lebedev over 3 years ago

Christian Kanzian wrote:

is all I get. Thats why I did "gdb darktable" and postet the output here. How do I install "debug symbols" then to get a bt?

https://wiki.debian.org/DebugPackage
https://wiki.debian.org/HowToGetABacktrace

#7 Updated by Christian Kanzian over 3 years ago

Kevin Gilbert wrote:

I too am suffering from this bug. Would it be possible to raise the priority?

Are you able to get a backtrace and can you attach it here?

Roman Lebedev wrote:

https://wiki.debian.org/HowToGetABacktrace

(gdb) symbol-file /opt/darktable/bin/darktable
Load new symbol table from "/opt/darktable/bin/darktable"? (y or n) y
Reading symbols from /opt/darktable/bin/darktable...done.
$ file /opt/darktable/bin/darktable
/opt/darktable/bin/darktable: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=97f7ad3810978d6c9bcd1baaf6c73660d2b23beb, not stripped

darktable has been build with debugging symbols?

I don't know what I'm doing wrong or what is missing - sorry. I only made updates via "apt-get update" and I'm not aware of any change on my system, which prevents the output of backtraces.

#8 Updated by Christian Kanzian about 3 years ago

Another trial with gdb.

#9 Updated by Ulrich Pegelow about 3 years ago

  • % Done changed from 20 to 10
  • Priority changed from Low to Critical
  • Status changed from Incomplete to Confirmed

Although my system doesn't crash I can confirm that there is really something wrong with the recent changes as far as OpenCL is concerned.

In my case I get black output in the [preview] pixelpipe while the [full] pixelpipe produces results as expected. I have two different GPUs in my system, AMD and NVIDIA, respectively. The problem is the same, regardless of which GPU I dedicate to which pixelpipe. So I assume that the differences between [preview] and [full] play a role.

darktable -d nan shows me that problems already occur in the first module:

[pixelpipe_process] [preview] using device 1
[dev_pixelpipe] took 0.000 secs (-0.000 CPU) initing base buffer [preview]
[dev_pixelpipe] took 0.000 secs (-0.000 CPU) processed `raw black/white point' on GPU, blended on GPU [preview]
[dev_pixelpipe] module `raw black/white point' min: (-0.066671; -0.066671; -0.066671) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.003 secs (0.003 CPU) processed `white balance' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `white balance' min: (-0.135991; -0.066671; -0.095591) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.001 secs (0.006 CPU) processed `highlight reconstruction' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `highlight reconstruction' min: (-0.135991; -0.066671; -0.095591) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.001 secs (0.000 CPU) processed `base curve' on GPU, blended on GPU [preview]
[dev_pixelpipe] module `base curve' min: (0.000000; 0.000000; 0.000000) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.001 secs (0.001 CPU) processed `input color profile' on GPU, blended on GPU [preview]
[dev_pixelpipe] module `input color profile' min: (0.000000; 0.000000; 0.000000) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.000 secs (0.001 CPU) processed `sharpen' on GPU, blended on GPU [preview]
[dev_pixelpipe] module `sharpen' min: (0.000000; 0.000000; 0.000000) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.001 secs (0.001 CPU) processed `output color profile' on GPU, blended on GPU [preview]
[dev_pixelpipe] module `output color profile' min: (0.000000; 0.000000; 0.000000) max: (0.000000; 0.000000; 0.000000) [preview]
[dev_pixelpipe] took 0.003 secs (0.004 CPU) processed `gamma' on CPU, blended on CPU [preview]

Here is what I would expect (from running [preview] of same image on CPU):

[pixelpipe_process] [preview] using device -1
[dev_pixelpipe] took 0.000 secs (0.000 CPU) initing base buffer [preview]
[dev_pixelpipe] took 0.001 secs (0.000 CPU) processed `raw black/white point' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `raw black/white point' min: (0.002536; 0.000000; 0.000000) max: (0.156011; 0.372928; 0.319855) [preview]
[dev_pixelpipe] took 0.001 secs (0.006 CPU) processed `white balance' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `white balance' min: (0.005172; 0.000000; 0.000000) max: (0.318220; 0.372928; 0.458600) [preview]
[dev_pixelpipe] took 0.002 secs (0.004 CPU) processed `highlight reconstruction' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `highlight reconstruction' min: (0.005172; 0.000000; 0.000000) max: (0.318220; 0.372928; 0.458600) [preview]
[dev_pixelpipe] took 0.001 secs (0.006 CPU) processed `base curve' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `base curve' min: (0.004807; 0.000000; 0.000000) max: (0.651138; 0.729813; 0.821503) [preview]
[dev_pixelpipe] took 0.001 secs (0.005 CPU) processed `input color profile' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `input color profile' min: (3.998910; -33.589706; -53.620453) max: (86.740974; 48.611103; 44.583874) [preview]
[dev_pixelpipe] took 0.002 secs (0.008 CPU) processed `sharpen' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `sharpen' min: (3.998910; -33.589706; -53.620453) max: (86.740974; 48.611103; 44.583874) [preview]
[dev_pixelpipe] took 0.002 secs (0.009 CPU) processed `output color profile' on CPU, blended on CPU [preview]
[dev_pixelpipe] module `output color profile' min: (0.073630; 0.000000; 0.000000) max: (0.798917; 0.841031; 0.924636) [preview]
[dev_pixelpipe] took 0.001 secs (0.003 CPU) processed `gamma' on CPU, blended on CPU [preview]

Wild guess: wrong OpenCL image buffer type given to the rawprepare module?

#10 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Although my system doesn't crash I can confirm that there is really something wrong with the recent changes as far as OpenCL is concerned.

In my case I get black output in the [preview] pixelpipe while the [full] pixelpipe produces results as expected.

Hm, i fixed [one of?] such case in the pr already
https://github.com/darktable-org/darktable/pull/1253/commits/a4f5c5fe2019fa34829a5dd4f39cd9771ea6b42d

I have two different GPUs in my system, AMD and NVIDIA, respectively. The problem is the same, regardless of which GPU I dedicate to which pixelpipe. So I assume that the differences between [preview] and [full] play a role.

darktable -d nan shows me that problems already occur in the first module:

[...]

Here is what I would expect (from running [preview] of same image on CPU):

[...]

Wild guess: wrong OpenCL image buffer type given to the rawprepare module?

I'll re-read code again, but without any opencl here, it may be complicated to troubleshoot.

#11 Updated by Roman Lebedev about 3 years ago

  • System changed from Debian to all

Yes, i know what's going on.

#12 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Although my system doesn't crash I can confirm that there is really something wrong with the recent changes as far as OpenCL is concerned.

In my case I get black output in the [preview] pixelpipe while the [full] pixelpipe produces results as expected. I have two different GPUs in my system, AMD and NVIDIA, respectively. The problem is the same, regardless of which GPU I dedicate to which pixelpipe. So I assume that the differences between [preview] and [full] play a role.

darktable -d nan shows me that problems already occur in the first module:

[...]

Here is what I would expect (from running [preview] of same image on CPU):

[...]

Wild guess: wrong OpenCL image buffer type given to the rawprepare module?

Better now?

#13 Updated by Roman Lebedev about 3 years ago

  • % Done changed from 10 to 100
  • Status changed from Confirmed to Fixed

#14 Updated by Ulrich Pegelow about 3 years ago

Looks like it's fixed now!

#15 Updated by Ulrich Pegelow about 3 years ago

  • % Done changed from 100 to 50
  • Status changed from Fixed to In Progress

I did some further tests and I think the ticket needs to be re-opened.

My observation of the preview window being black is fixed. However, I can confirm the previous reports about crashes with OpenCL.

Here are the steps to provoke:

1) open an image in darkroom
2) zoom into the image (wheel scroll) and zoom out again
3) move a slider (e.g. exposure)
--> segmentation fault without any backtrace

I have two GPUs which I tested (AMD and NVIDIA). Both show the same behavior. It is linked to the [full] pixelpipe being processed with OpenCL.

#16 Updated by Ulrich Pegelow about 3 years ago

Additional observations from running in gdb. The crash is related to a memcpy_sse3 within the OpenCL driver library triggered in pixelpipe_hb.c:1544. That code part comes into play when a module's parameters cause the module not to run in OpenCL (piece->process_cl_ready = 0). In my case it's in the temperature IOP with an x-trans image.

#17 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Additional observations from running in gdb. The crash is related to a memcpy_sse3 within the OpenCL driver library triggered in pixelpipe_hb.c:1544. That code part comes into play when a module's parameters cause the module not to run in OpenCL (piece->process_cl_ready = 0). In my case it's in the temperature IOP with an x-trans image.

Can you build dt in debug build mode (LDFLAGS="-O0 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-common -U_FORTIFY_SOURCE" CFLAGS="${LDFLAGS}" CXXFLAGS="${LDFLAGS}" CPPFLAGS="${LDFLAGS}" CC=gcc-6 CXX=g++-6 cmake -DCMAKE_BUILD_TYPE=Debug ../)

let it crash, and show all the console output and the backtrace?

#18 Updated by Ulrich Pegelow about 3 years ago

I did like proposed (even upgraded to gcc6) but to no avail. It crashes with the following output:

pegelow@zaphod:~> /usr/local/test/bin/darktable
Speicherzugriffsfehler

"Speicherzugriffsfehler" is German for Segfault. Nothing more, no backtrace.

#19 Updated by Dirk Sagwitz about 3 years ago

For me it's still crashing, too. Not sure whether it can help you, but I'll attach the dump which was written during the crash.
(btw. the bug was introduced after commit abf298cbd44135c147fd3cb71caaedf808e36d1a, building this version everything is working fine)

#20 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

I did like proposed (even upgraded to gcc6) but to no avail. It crashes with the following output:

[...]

"Speicherzugriffsfehler" is German for Segfault. Nothing more, no backtrace.

Yeah, that happens sometimes.
Just do the usual dance, something like
$ gdb darktable

run
set logging file gdb.txt
set logging redirect on
where
thread apply all bt full

#21 Updated by Roman Lebedev about 3 years ago

Dirk Sagwitz wrote:

For me it's still crashing, too. Not sure whether it can help you, but I'll attach the dump which was written during the crash.
(btw. the bug was introduced after commit abf298cbd44135c147fd3cb71caaedf808e36d1a, building this version everything is working fine)

Bt is a bit useless to be i'm afraid.
Could you please try compiling dt like this (this is different from what i suggested to Ulrich):

cd ~/darktable/build/ && rm -rf * && LDFLAGS="-O0 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-common -U_FORTIFY_SOURCE -fsanitize=address -fstack-protector-strong" CFLAGS="${LDFLAGS}" CXXFLAGS="${LDFLAGS}" CPPFLAGS="${LDFLAGS}" CC=gcc-6 CXX=g++-6 cmake -DCMAKE_BUILD_TYPE=Debug ../ && make -j9 && sudo make -j9 install

And then start darktable from console, make it crash, and show all the console output.
Maybe we're lucky and it will cause ASan to catch this memory issue.

#22 Updated by Ulrich Pegelow about 3 years ago

Debug session attached.

#23 Updated by Dirk Sagwitz about 3 years ago

I messed with two parallel installations ... will do t again. Is there a chance to setup those compile parameters in the build.sh?

#24 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Debug session attached.

Hmm, thank you for that backtrace.
But unfortunately now i'm completely lost.

Could you maybe try the compilation line i suggested in https://redmine.darktable.org/issues/11149#note-21, maybe it will work for you?

#25 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

Could you maybe try the compilation line i suggested in https://redmine.darktable.org/issues/11149#note-21, maybe it will work for you?

With that setting I get "No stack".

#26 Updated by Roman Lebedev about 3 years ago

But does it still use opencl and crash?
With that setting, just start dt from console, without gdb, and just show the console output.

#27 Updated by Ulrich Pegelow about 3 years ago

It uses OpenCL and it crashes. Here is the console output:

[dev_pixelpipe] module `Entrastern' min: (0,000068; 0,000000; 0,000000) max: (0,004942; 0,004060; 0,004077) [thumbnail]
[dev_pixelpipe] module `Basiskurve' min: (0,000061; 0,000000; 0,000000) max: (0,004623; 0,003876; 0,003876) [thumbnail]
[dev_pixelpipe] module `Eingabefarbprofil' min: (0,048058; -0,547171; -1,429492) max: (3,679546; 2,128936; 2,177569) [thumbnail]
[dev_pixelpipe] module `Schärfen' min: (0,048058; -0,547171; -1,429492) max: (3,679546; 2,128936; 2,177569) [thumbnail]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,010638; 0,000000; 0,000000) max: (0,088035; 0,079628; 0,081196) [thumbnail]
[dev_pixelpipe] module `Entrastern' min: (0,001576; -0,007899; -0,011605) max: (0,329805; 0,387420; 0,468292) [full]
[dev_pixelpipe] module `Basiskurve' min: (0,001587; 0,000000; 0,000000) max: (0,669052; 0,747955; 0,829727) [full]
[dev_pixelpipe] module `Eingabefarbprofil' min: (0,333984; -29,709457; -53,682381) max: (87,774361; 53,461029; 52,206005) [full]
[dev_pixelpipe] module `Schärfen' min: (-1,024647; -29,709457; -53,682381) max: (87,774361; 53,461029; 52,206005) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,014710; 0,000000; 0,000000) max: (0,809964; 0,854200; 0,928191) [full]
[dev_pixelpipe] module `Raw-Schwarz-/Weißpunkt' min: (0,002536; 0,000000; 0,000000) max: (0,156011; 0,372928; 0,319855) [preview]
[dev_pixelpipe] module `Weißabgleich' min: (0,005172; 0,000000; 0,000000) max: (0,318220; 0,372928; 0,458600) [preview]
[dev_pixelpipe] module `Spitzlicht-Rekonstruktion' min: (0,005172; 0,000000; 0,000000) max: (0,318220; 0,372928; 0,458600) [preview]
[dev_pixelpipe] module `Basiskurve' min: (0,004807; 0,000000; 0,000000) max: (0,651138; 0,729813; 0,821503) [preview]
[dev_pixelpipe] module `Eingabefarbprofil' min: (3,998910; -33,589706; -53,620453) max: (86,740974; 48,611103; 44,583870) [preview]
[dev_pixelpipe] module `Schärfen' min: (3,998910; -33,589706; -53,620453) max: (86,740974; 48,611103; 44,583870) [preview]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,073630; 0,000000; 0,000000) max: (0,798917; 0,841031; 0,924636) [preview]
[dev_pixelpipe] module `Entrastern' min: (0,003013; 0,000000; 0,000000) max: (0,334212; 0,392206; 0,462065) [full]
[dev_pixelpipe] module `Basiskurve' min: (0,002930; 0,000000; 0,000000) max: (0,675705; 0,753693; 0,824493) [full]
[dev_pixelpipe] module `Eingabefarbprofil' min: (2,857872; -29,228867; -38,667362) max: (88,104706; 40,368126; 64,582703) [full]
[dev_pixelpipe] module `Schärfen' min: (0,760377; -29,228867; -38,667362) max: (88,354263; 40,368126; 64,582703) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,046021; 0,000000; 0,000000) max: (0,816876; 0,861357; 0,933730) [full]
[dev_pixelpipe] module `Entrastern' min: (0,004087; 0,000000; 0,000000) max: (0,347325; 0,402882; 0,466701) [full]
[dev_pixelpipe] module `Basiskurve' min: (0,003876; 0,000000; 0,000000) max: (0,694901; 0,766037; 0,828415) [full]
[dev_pixelpipe] module `Eingabefarbprofil' min: (2,673788; -30,933455; -45,595043) max: (88,802582; 37,714870; 63,070858) [full]
[dev_pixelpipe] module `Schärfen' min: (-1,710919; -30,933455; -45,595043) max: (93,195404; 37,714870; 63,070858) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,000000; 0,000000; 0,000000) max: (0,876646; 0,939071; 0,944839) [full]
[dev_pixelpipe] module `Entrastern' min: (0,001576; -0,007899; -0,011605) max: (0,329805; 0,387420; 0,468292) [full]
[dev_pixelpipe] module `Basiskurve' min: (0,001587; 0,000000; 0,000000) max: (0,669052; 0,747955; 0,829727) [full]
[dev_pixelpipe] module `Eingabefarbprofil' min: (0,333984; -29,709457; -53,682381) max: (87,774361; 53,461029; 52,206005) [full]
[dev_pixelpipe] module `Schärfen' min: (-1,024647; -29,709457; -53,682381) max: (87,774361; 53,461029; 52,206005) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,014710; 0,000000; 0,000000) max: (0,809964; 0,854200; 0,928191) [full]
[dev_pixelpipe] module `Raw-Schwarz-/Weißpunkt' min: (0,002536; 0,000000; 0,000000) max: (0,156011; 0,372928; 0,319855) [preview]
[dev_pixelpipe] module `Weißabgleich' min: (0,005172; 0,000000; 0,000000) max: (0,318220; 0,372928; 0,458600) [preview]
[dev_pixelpipe] module `Spitzlicht-Rekonstruktion' min: (0,005172; 0,000000; 0,000000) max: (0,318220; 0,372928; 0,458600) [preview]
[dev_pixelpipe] module `Belichtung' min: (0,004989; 0,000000; 0,000000) max: (0,306955; 0,359726; 0,442364) [preview]
[dev_pixelpipe] module `Basiskurve' min: (0,004654; 0,000000; 0,000000) max: (0,633102; 0,712250; 0,806824) [preview]
[dev_pixelpipe] module `Eingabefarbprofil' min: (3,879010; -33,532516; -52,965134) max: (85,839767; 48,117561; 43,780220) [preview]
[dev_pixelpipe] module `Schärfen' min: (3,879010; -33,532516; -52,965134) max: (85,839767; 48,117561; 43,780220) [preview]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,072597; 0,000000; 0,000000) max: (0,788458; 0,830640; 0,917708) [preview]
[dev_pixelpipe] module `Entrastern' min: (0,001576; -0,007899; -0,011605) max: (0,329805; 0,387420; 0,468292) [full]
[dev_pixelpipe] module `Belichtung' min: (0,004072; 0,000000; 0,000000) max: (0,250537; 0,293609; 0,361059) [preview]
[dev_pixelpipe] module `Belichtung' min: (0,001520; -0,007620; -0,011194) max: (0,318129; 0,373705; 0,451714) [full]
[dev_pixelpipe] module `Basiskurve' min: (0,003876; 0,000000; 0,000000) max: (0,534515; 0,610977; 0,714081) [preview]
[dev_pixelpipe] module `Eingabefarbprofil' min: (3,207013; -32,256008; -48,240929) max: (80,460503; 43,815388; 38,017479) [preview]
[dev_pixelpipe] module `Basiskurve' min: (0,001526; 0,000000; 0,000000) max: (0,650986; 0,730804; 0,815414) [full]
[dev_pixelpipe] module `Schärfen' min: (3,207013; -32,256008; -48,240929) max: (80,460503; 43,815388; 38,017479) [preview]
[dev_pixelpipe] module `Eingabefarbprofil' min: (0,321138; -29,271498; -53,543602) max: (86,907776; 52,938625; 50,649429) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,066362; 0,000000; 0,000000) max: (0,730434; 0,773480; 0,870039) [preview]
[dev_pixelpipe] module `Schärfen' min: (-1,010918; -29,271498; -53,543602) max: (86,907776; 52,938625; 50,649429) [full]
[dev_pixelpipe] module `Ausgabefarbprofil' min: (0,011765; 0,000000; 0,000000) max: (0,799451; 0,844114; 0,921492) [full]
=================================================================
==751==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f9f4edd7820 at pc 0x7f9fc714f80f bp 0x7f9fa5aa17e0 sp 0x7f9fa5aa0f90
READ of size 79872 at 0x7f9f4edd7820 thread T10
    #0 0x7f9fc714f80e  (/usr/lib64/libasan.so.3+0x5f80e)
    #1 0x7f9fa9734687  (/usr/lib64/libamdocl64.so+0x6e3687)
    #2 0x7f9fa9766d62  (/usr/lib64/libamdocl64.so+0x715d62)
    #3 0x7f9fa979b01a  (/usr/lib64/libamdocl64.so+0x74a01a)
    #4 0x7f9fa9742dac  (/usr/lib64/libamdocl64.so+0x6f1dac)
    #5 0x7f9fa974312c  (/usr/lib64/libamdocl64.so+0x6f212c)
    #6 0x7f9fa96e115e  (/usr/lib64/libamdocl64.so+0x69015e)
    #7 0x7f9fa975115b  (/usr/lib64/libamdocl64.so+0x70015b)
    #8 0x7f9fc28890a3 in start_thread (/lib64/libpthread.so.0+0x80a3)
    #9 0x7f9fbec82cbc in __clone (/lib64/libc.so.6+0xe4cbc)

0x7f9f4edd7820 is located 0 bytes to the right of 32907296-byte region [0x7f9f4ce75800,0x7f9f4edd7820)
allocated by thread T0 here:
    #0 0x7f9fc71b5710 in __interceptor_posix_memalign (/usr/lib64/libasan.so.3+0xc5710)
    #1 0x7f9fc68ea36f in dt_alloc_align /home/pegelow/darktable/src/common/darktable.c:1181
    #2 0x7f9fc69802cb in dt_mipmap_cache_alloc /home/pegelow/darktable/src/common/mipmap_cache.c:238
    #3 0x7f9fc6968114 in dt_imageio_open_rawspeed /home/pegelow/darktable/src/common/imageio_rawspeed.cc:314
    #4 0x7f9fc69545a2 in dt_imageio_open /home/pegelow/darktable/src/common/imageio.c:953
    #5 0x7f9fc6984348 in dt_mipmap_cache_get_with_caller /home/pegelow/darktable/src/common/mipmap_cache.c:702
    #6 0x7f9fc69ee91b in _dt_dev_load_raw /home/pegelow/darktable/src/develop/develop.c:422
    #7 0x7f9fc69ef2d9 in dt_dev_load_image /home/pegelow/darktable/src/develop/develop.c:472
    #8 0x7f9f9aa27e12 in enter /home/pegelow/darktable/src/views/darkroom.c:1631
    #9 0x7f9fc6ba7174 in dt_view_manager_switch /home/pegelow/darktable/src/views/view.c:462
    #10 0x7f9fc69cc2a5 in _dt_ctl_switch_mode_to /home/pegelow/darktable/src/control/control.c:453
    #11 0x7f9fc642e904 in g_main_context_invoke_full (/usr/lib64/libglib-2.0.so.0+0x4c904)

Thread T10 created by T0 here:
    #0 0x7f9fc7123f09 in __interceptor_pthread_create (/usr/lib64/libasan.so.3+0x33f09)
    #1 0x7f9fa975188b  (/usr/lib64/libamdocl64.so+0x70088b)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib64/libasan.so.3+0x5f80e) 
Shadow bytes around the buggy address:
  0x0ff469db2eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff469db2ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff469db2ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff469db2ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff469db2ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ff469db2f00: 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa fa
  0x0ff469db2f10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff469db2f20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff469db2f30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff469db2f40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff469db2f50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==751==ABORTING

#28 Updated by Roman Lebedev about 3 years ago

Just to be sure, it does not happen with 4217391301f0bc81a83746af8981ce4a1c5738b7 ?
I.e. it is not a bug in amd blob?

#29 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

Just to be sure, it does not happen with 4217391301f0bc81a83746af8981ce4a1c5738b7 ?

Confirmed, with that commit I don't get a crash. IMHO it's the commit following (inclusion of PR1253) where the problem starts.

I.e. it is not a bug in amd blob?

The crash happens with both, AMD and with NVIDIA GPUs as soon as they process the [full] pixelpipe (FYI: I have both ones and can dedicate them with opencl_device_priority). This is tested with my default settings; with the debug settings I can't confirm accurately as my NVIDIA card is not detected. However, I assume that the problem does not lie in the individual drivers.

#30 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Roman Lebedev wrote:

Just to be sure, it does not happen with 4217391301f0bc81a83746af8981ce4a1c5738b7 ?

Confirmed, with that commit I don't get a crash.

OK.

IMHO it's the commit following (inclusion of PR1253) where the problem starts.

Then may i ask you to bisect it between (edited to use proper commit ids)
good 829fb7d7fdd03b56d96e32ba9562fe0fedd403ba
bad a4f5c5fe2019fa34829a5dd4f39cd9771ea6b42d
?

Maybe that will help narrow it down, or maybe it won't.

I.e. it is not a bug in amd blob?

The crash happens with both, AMD and with NVIDIA GPUs as soon as they process the [full] pixelpipe (FYI: I have both ones and can dedicate them with opencl_device_priority). This is tested with my default settings; with the debug settings I can't confirm accurately as my NVIDIA card is not detected. However, I assume that the problem does not lie in the individual drivers.

#31 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

Then may i ask you to bisect it between (edited to use proper commit ids)
good 829fb7d7fdd03b56d96e32ba9562fe0fedd403ba
bad a4f5c5fe2019fa34829a5dd4f39cd9771ea6b42d
?

Maybe that will help narrow it down, or maybe it won't.

Well, that has been one of the first things I tried when seeing the issue. However, numerous in-between commits don't compile here due to some Exiv2 problem ("has no member named 'select_node'"). One can try to 'git bisect skip' those but this makes bisect loose track completely. Non-compiling commits are a real problem here.

#32 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Roman Lebedev wrote:

Then may i ask you to bisect it between (edited to use proper commit ids)
good 829fb7d7fdd03b56d96e32ba9562fe0fedd403ba
bad a4f5c5fe2019fa34829a5dd4f39cd9771ea6b42d
?

Maybe that will help narrow it down, or maybe it won't.

Well, that has been one of the first things I tried when seeing the issue. However, numerous in-between commits don't compile here due to some Exiv2 problem ("has no member named 'select_node'"). One can try to 'git bisect skip' those but this makes bisect loose track completely. Non-compiling commits are a real problem here.

You can fix those by applying https://github.com/darktable-org/darktable/commit/97868783c1d53672ef1a5b2acfe7f3b3d0abd851.patch

#33 Updated by Ulrich Pegelow about 3 years ago

Here is the bisect result:

a18488567cd2a1072f8516518eba0f1b5f612351 is the first bad commit
commit a18488567cd2a1072f8516518eba0f1b5f612351
Author: Roman Lebedev <lebedev.ri@gmail.com>
Date:   Wed Aug 31 21:12:37 2016 +0300

    IOPs, pipe: replace output_bpp() with output_format()

    If output_bpp() is the same as default-one - drop it

:040000 040000 745685c2da680fff1a03d32130f78a0801f56ab3 c4370c8cf12e607a4cd96c10ed2e2cbf1d24f7ed M      src

#34 Updated by Dirk Sagwitz about 3 years ago

Unfortunately I did not find a way yet to get debugging and opencl running in parallel. I am using an Asus laptop with NVidia / primus to run darktable, but when I compile darktable with debugging option opencl is not found :-(

#35 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

It uses OpenCL and it crashes. Here is the console output:

[...]

Ok, this makes no sense whatsoever.
If it is pixelpipe_hb.c:1544, then why it reads the host buffer (as opposed to writing to it) ?
If this is temperature.c, then why the host buffer is the original uint16_t that was returned by rawspeed and was the input into rawprepare.c ?

#36 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

Ok, this makes no sense whatsoever.
If it is pixelpipe_hb.c:1544, then why it reads the host buffer (as opposed to writing to it) ?
If this is temperature.c, then why the host buffer is the original uint16_t that was returned by rawspeed and was the input into rawprepare.c ?

Background is as follows. The OpenCL pixelpipe tries to keep all buffers on the GPU as far as possible because any transfer between host and device is very expensive. Therefore the output buffer of a first module is directly taken as input buffer of the second module etc. We normally never copy back intermediate output to the host (except if opencl_synch_cache is set to TRUE). Now what happens if a module is not ready to run on OpenCL (e.g. a module has process_cl() but piece->process_cl_ready = 0 tells that it needs to be processed on CPU)? In that case the OpenCL input buffer of that module needs first to be copied back to host memory. Of course input and output buffers of temperature.c must be float (1 channel in case of a raw).

#37 Updated by Roman Lebedev about 3 years ago

  • Has duplicate Bug #11158: dt crash badly when working interactively added

#38 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Roman Lebedev wrote:

Ok, this makes no sense whatsoever.
If it is pixelpipe_hb.c:1544, then why it reads the host buffer (as opposed to writing to it) ?
If this is temperature.c, then why the host buffer is the original uint16_t that was returned by rawspeed and was the input into rawprepare.c ?

Background is as follows. The OpenCL pixelpipe tries to keep all buffers on the GPU as far as possible because any transfer between host and device is very expensive. Therefore the output buffer of a first module is directly taken as input buffer of the second module etc. We normally never copy back intermediate output to the host (except if opencl_synch_cache is set to TRUE). Now what happens if a module is not ready to run on OpenCL (e.g. a module has process_cl() but piece->process_cl_ready = 0 tells that it needs to be processed on CPU)? In that case the OpenCL input buffer of that module needs first to be copied back to host memory. Of course input and output buffers of temperature.c must be float (1 channel in case of a raw).

Thank you for the information, but unfortunately i already knew that, and it matches my memory of how it works exactly :(

#39 Updated by Roman Lebedev about 3 years ago

Hm, Pascal's bt is different.
It does not seem to crash in pixelpipe_hb.c:1544

#40 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

Hm, Pascal's bt is different.
It does not seem to crash in pixelpipe_hb.c:1544

Yes, it seems to crash in pixelpipe_hb.c:1052 (around line 1915 in the debug output). I can't tell which module the pixelpipe runs here but there is one thing that is interesting. Width and height of the input image is 5920 and 3941, respectively, with in_bpp=4. The buffer size of the output buffer is 89436184, however (bufsize variable). As long as that modules does not have a roi_out that is smaler than roi_in, shouldn't bufsize not be 93322880?

(side remark: for debugging purposes it would make sense to somehow get the string of module->op duplicated into one local variable in dt_dev_pixelpipe_process_rec())

#41 Updated by Ulrich Pegelow about 3 years ago

Got some interesting insight into the problem. I added a debug output close to pixelpipe_hb.c:1044, i.e. exactly at the place where new OpenCL input buffers get allocated. This step occurs whenever an OpenCL-enabled module finds its input on host memory:

allocate opencl input buffer for module 'rawprepare' in [thumbnail] with: width 4992, height 3296, bpp 2
allocate opencl input buffer for module 'basecurve' in [thumbnail] with: width 1348, height 900, bpp 16
allocate opencl input buffer for module 'rawprepare' in [thumbnail] with: width 5000, height 3296, bpp 2
allocate opencl input buffer for module 'demosaic' in [thumbnail] with: width 4944, height 3296, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 4992, height 3296, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 4936, height 3296, bpp 4
allocate opencl input buffer for module 'rawprepare' in [preview] with: width 681, height 450, bpp 16
allocate opencl input buffer for module 'basecurve' in [preview] with: width 673, height 450, bpp 16
allocate opencl input buffer for module 'rawprepare' in [full] with: width 3497, height 3113, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 3441, height 3113, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 1612, height 1408, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 1556, height 1408, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 1425, height 1238, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 1369, height 1238, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 980, height 836, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 924, height 836, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 1277, height 1105, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 1221, height 1105, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 3497, height 3113, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 3441, height 3113, bpp 4
allocate opencl input buffer for module 'rawprepare' in [full] with: width 4992, height 3296, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 4936, height 3296, bpp 4
allocate opencl input buffer for module 'rawprepare' in [preview] with: width 681, height 450, bpp 16
allocate opencl input buffer for module 'rawprepare' in [full] with: width 4992, height 3296, bpp 2
allocate opencl input buffer for module 'demosaic' in [full] with: width 4936, height 3296, bpp 4
allocate opencl input buffer for module 'exposure' in [preview] with: width 673, height 450, bpp 16
allocate opencl input buffer for module 'rawprepare' in [full] with: width 4992, height 3296, bpp 16

Look at the last line which is generated immediately before the crash happens. Module 'rawprepare' allocates an OpenCL input buffer with bpp 16. That's of cause wrong as the input image is a raw file it should be 2 as in the cases before. Now OpenCL will happily allocate that buffer. However, it is much too big. When at a later time that buffer needs to be copied back to host memory (in cases as discussed before, e.g. in line 1544) the host "shadow" buffer is too small and we get a segfault.

@Roman: assuming that rawprepare should ask for bpp 2, why does it ask for bpp 16 in some cases?

As far as I can see only rawprepare causes these issues. If the real cause is not found easily, there would be a quick fix by disabling OpenCL in rawprepare for the moment (until the reason is found).

#42 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Got some interesting insight into the problem. I added a debug output close to pixelpipe_hb.c:1044, i.e. exactly at the place where new OpenCL input buffers get allocated. This step occurs whenever an OpenCL-enabled module finds its input on host memory:

[...]

Look at the last line which is generated immediately before the crash happens. Module 'rawprepare' allocates an OpenCL input buffer with bpp 16. That's of cause wrong as the input image is a raw file it should be 2 as in the cases before. Now OpenCL will happily allocate that buffer. However, it is much too big. When at a later time that buffer needs to be copied back to host memory (in cases as discussed before, e.g. in line 1544) the host "shadow" buffer is too small and we get a segfault.

@Roman: assuming that rawprepare should ask for bpp 2, why does it ask for bpp 16 in some cases?

As far as I can see only rawprepare causes these issues. If the real cause is not found easily, there would be a quick fix by disabling OpenCL in rawprepare for the moment (until the reason is found).

I guess the wrong strut is being used somewhere.
I'm adding a sanity check, hopefully it will catch it.

#43 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Got some interesting insight into the problem. I added a debug output close to pixelpipe_hb.c:1044, i.e. exactly at the place where new OpenCL input buffers get allocated. This step occurs whenever an OpenCL-enabled module finds its input on host memory:

[...]

Look at the last line which is generated immediately before the crash happens. Module 'rawprepare' allocates an OpenCL input buffer with bpp 16. That's of cause wrong as the input image is a raw file it should be 2 as in the cases before. Now OpenCL will happily allocate that buffer. However, it is much too big. When at a later time that buffer needs to be copied back to host memory (in cases as discussed before, e.g. in line 1544) the host "shadow" buffer is too small and we get a segfault.

@Roman: assuming that rawprepare should ask for bpp 2, why does it ask for bpp 16 in some cases?

As far as I can see only rawprepare causes these issues. If the real cause is not found easily, there would be a quick fix by disabling OpenCL in rawprepare for the moment (until the reason is found).

@Ulrich does it still crash with current git master?

#44 Updated by Ulrich Pegelow about 3 years ago

Roman Lebedev wrote:

@Ulrich does it still crash with current git master?

Unfortunately, yes. With and without the debug output. After some time of trying I get:

allocate opencl input buffer for module 'rawprepare' in [full] with: width 4992, height 3296, bpp 16

and a crash.

#45 Updated by Ulrich Pegelow about 3 years ago

Addition: bpp=16 happens in the [full] pixelpipe despite the fact that with Jo's recent changes rawprepare uses bpp=2 also in the [preview] pixelpipe. Strange.

#46 Updated by Roman Lebedev about 3 years ago

Ulrich Pegelow wrote:

Addition: bpp=16 happens in the [full] pixelpipe

Now actually reproduced.

despite the fact that

with Jo's recent changes rawprepare uses bpp=2 also in the [preview] pixelpipe.

Hm :)

Strange.

#47 Updated by Roman Lebedev about 3 years ago

  • % Done changed from 50 to 20
  • Status changed from In Progress to Triaged

Ok, i guess i understand what is going on.
Will fix tomorrow.

#48 Updated by Albert Castells about 3 years ago

Hi everyone. My DT build also crashes when using opencl in darkroom (not when exporting). I just need to crop the image to make it crash. Attached is the backtrace...

#49 Updated by Roman Lebedev about 3 years ago

  • % Done changed from 20 to 70
  • Status changed from Triaged to Patch attached

Please try https://github.com/LebedevRI/darktable/tree/opencl-crash

This issue is due to https://github.com/darktable-org/darktable/blob/b83264f7be9d668ad2e0d8b764991a3d151847c7/src/develop/pixelpipe_hb.c#L798-L840,
in particular due to **out_format = pipe->dsc;
Sometimes that just happens to contain the buffer description of "gamma" out buffer.
I can not figure out how to fix it the right way.

Assuming the fix works, we could live with that for now.

#50 Updated by Pascal Obry about 3 years ago

@Roman: Just tested your patch and it fixes the issue on my side. No more crash.

#51 Updated by Ulrich Pegelow about 3 years ago

Looks good.

#52 Updated by Roman Lebedev about 3 years ago

Okay, i'll push this 'patch'.
But this must be fixed properly.

#53 Updated by Christian Kanzian about 3 years ago

Todays git master and OpenCL works for me again.

Thank you very much!

#54 Updated by Roman Lebedev about 3 years ago

  • % Done changed from 70 to 100
  • Status changed from Patch attached to Fixed

Not sure why redmine still did not pull new commits and close this automatically.

#55 Updated by Roman Lebedev about 3 years ago

  • Precedes Bug #11168: Recent pipe changes: buffer dsc caching issues added

#56 Updated by Roman Lebedev about 3 years ago

Please check whether my last 3 commits reintroduced the problem or not.
If not, i guess this may be the mostly final fix :)

#57 Updated by Pascal Obry about 3 years ago

All is fine for me with current master. Great work Roman!

#58 Updated by Ulrich Pegelow about 3 years ago

Also on my side all fine.

#59 Updated by Christian Kanzian about 3 years ago

Fine here too!

#60 Updated by Dirk Sagwitz about 3 years ago

On my side its fine, too. Congrats!

#61 Updated by Roman Lebedev about 3 years ago

  • Target version set to 2.2.0

Also available in: Atom PDF

Go to top