Mac: OpenCL kernels compilation sporadically complain about missing common.h
The macOS app is missing the file source:data/kernels/common.h which is supposed to be located at
darktable.app/Contents/Resources/share/darktable/kernels/common.h, together with all the other OpenCL kernels. Therefore, some OpenCL kernels fail to build and OpenCL is globally deactivated in darktable.
Solution: Put source:data/kernels/common.h into
I was able to fix OpenCL support locally by putting the file there. But you should fix your build system as well.
#2 Updated by Igor Kuzmin over 3 years ago
- % Done changed from 0 to 20
- Status changed from New to Incomplete
I have no idea what are you talking about, "make install" installs common.h in correct path and DMG contains it also. OpenCL works for me.
How exactly did you manage to come up with darktable.app without that file?
#3 Updated by Christian Stussak over 3 years ago
- File darktable_opencl_3rd_run.log darktable_opencl_3rd_run.log added
- File darktable_opencl_2nd_run.log darktable_opencl_2nd_run.log added
- File darktable_opencl_1st_run.log darktable_opencl_1st_run.log added
I thought this issue would be trivial to track down and resolve, but I was wrong. Also, my initial analysis was too brief and wrong. It doesn't seem to be related to the build system. I downloaded https://github.com/darktable-org/darktable/releases/download/release-2.2.1/darktable-184.108.40.206.dmg again and the file
common.h is there like it should.
Actually, I never checked if it was ever missing before. darktable complained that it could not find the file. I copied it over (via command line) and restarted darktable. OpenCL was working again. After your comment, I was about to mark this issue as invalid.
However, this issue was a mystery to me (I also felt kind of stupid ...) and digged deeper into it.
rm -r ~/.cache/darktable/cached_kernels_for_*, I did
/Applications/darktable.app/Contents/MacOS/darktable -d opencl and voilà, it complained about the missing
I did this a couple of times: delete the OpenCL cache and restart darktable. The affected kernels sometimes where the same, sometimes they changed. After a couple of runs of darktable without deleting the cache, darktable finally succeeded to compile and cache all of the OpenCL kernels. Then I deleted the cache, started dt again: it worked on the first run. Next round: It took 6 attempts to compile all shaders correctly.
'common.h' not found issue is triggered non-deterministic, my impression is that this is related to a race condition. Maybe something related to file locking of
common.h? (just guessing)
I attached the logs for three consecutive runs of
/Applications/darktable.app/Contents/MacOS/darktable -d opencl. I deleted the cache only before the first run. Here are the essential parts:
[opencl_init] compiling program `nlmeans.cl' .. [opencl_fopen_stat] could not open file `/Users/stussak/.cache/darktable/cached_kernels_for_HDGraphics4000/nlmeans.cl.bin'! [opencl_load_program] could not load cached binary program, trying to compile source [opencl_load_program] successfully loaded program from `/Applications/darktable.app/Contents/Resources/share/darktable/kernels/nlmeans.cl' [opencl_build_program] could not build program: -11 [opencl_build_program] BUILD STATUS: -2 BUILD LOG: <program source>:20:10: fatal error: 'common.h' file not found #include "common.h" ^ [opencl_init] failed to compile program `nlmeans.cl'! [opencl_init] FINALLY: opencl is NOT AVAILABLE on this system. [opencl_init] initial status of opencl enabled flag is OFF.
[opencl_init] compiling program `basecurve.cl' .. [opencl_fopen_stat] could not open file `/Users/stussak/.cache/darktable/cached_kernels_for_HDGraphics4000/basecurve.cl.bin'! [opencl_load_program] could not load cached binary program, trying to compile source [opencl_load_program] successfully loaded program from `/Applications/darktable.app/Contents/Resources/share/darktable/kernels/basecurve.cl' [opencl_build_program] could not build program: -11 [opencl_build_program] BUILD STATUS: -2 BUILD LOG: <program source>:20:10: fatal error: 'common.h' file not found #include "common.h" ^ [opencl_init] failed to compile program `basecurve.cl'! [opencl_init] FINALLY: opencl is NOT AVAILABLE on this system. [opencl_init] initial status of opencl enabled flag is OFF.
... compiling kernels ... [opencl_init] kernel loading time: 5.3044 [opencl_init] OpenCL successfully initialized. [opencl_init] here are the internal numbers and names of OpenCL devices available to darktable: [opencl_init] 0 'HD Graphics 4000' [opencl_init] 1 'GeForce GT 650M' [opencl_init] these are your device priorities: [opencl_init] image preview export thumbnail [opencl_init] 0 1 0 0 [opencl_init] 1 -1 1 1 [opencl_init] FINALLY: opencl is AVAILABLE on this system. [opencl_init] initial status of opencl enabled flag is ON.
#5 Updated by Igor Kuzmin over 3 years ago
- % Done changed from 20 to 10
- Assignee deleted (
- Status changed from Incomplete to Confirmed
- Category changed from Buildsystem to General
- Subject changed from OSX DMG: OpenCL support broken due to missing common.h to Mac: OpenCL kernels compilation sporadically complain about missing common.h
Confirmed, but I have no idea why it's happening (or did it actually start with 2.2 release). Include dir is specified as compiler option (-I) for every kernel, I can't see any race condition. I would consider that as one of the many OpenCL issues on Mac. I've left the issue opened, but removed myself from assignees, because I won't investigate further.
#6 Updated by Christian Stussak over 3 years ago
I can't tell wether it started with 2.2.0. Probably not. But during the past days, I was digging into a couple of issues I experienced with darktable recently (I finally had time for that). Usually, I just start darktable through the menu, but for the investigations, I started darktable from the command line (
-d all) and spotted the OpenCL problems.
I just checked (via
fs_usage) which files are accessed by the OpenCL compiler. I expected to see an error related to access restrictions or file not found for
.../share/darktable/kernels/common.h. Instead, I got this surprising result: During the run where the compilation fails, the OpenCL compiler does not even attempt to access the correct file. It just ignores the path given by the
-I option (which is correctly given, I checked this as well). Instead, it tries to open
common.h in various system path know to the OpenCL compiler and, of course, can't find anything (
ENOENT). It is using the correct
.../share/darktable/kernels/common.h path for all other runs.
So, this looks like an issue with the OpenCL compiler on macOS (I am on 10.11.6).
Luckily, we have a workaround: Restart darktable a couple of times. It will eventually succeed compiling all OpenCL kernels. Due to the cache for compiled kernels, you only need to do this after each program update. May this serve someone who runs into the same issue, I ran into.
#7 Updated by Jilles van Gurp over 1 year ago
I'm running into this with a recent macbook pro with a AMD Radeon Pro 560 on both 2.4.4 and 2.6.0Rc2. This is still an issue and I can confirm the workaround of starting multiple times still works. Thanks Christian!
After reading this thread, I verified the problem with ./darktable-cltest. It indeed complains about common.h
My imac 5k used to have the same issue and started working after mojave 10.14.1 a few weeks ago. I first blamed this on a driver issue that apparently got fixed but after reading this thread, I decided to give restarting darktable a few times a try on my mac book.
A few observations that may be helpful:
- the number of files in .cache grows between restarts. It is definitely succeeding at building progressively more. This means there definitely is some issue with the build.
- if fails for both the intel 630 and amd 560 initially. Judging from the file names, it actually gives up before even trying to build the files for the 560 the first time.
- at some point the compilation succeeds and I can turn on opencl
I hope this helps.