Project

General

Profile

Bug #11423

Mac: OpenCL kernels compilation sporadically complain about missing common.h

Added by Christian Stussak over 3 years ago. Updated over 1 year ago.

Status:
Confirmed
Priority:
Low
Assignee:
-
Category:
General
Target version:
-
Start date:
01/03/2017
Due date:
% Done:

10%

Estimated time:
Affected Version:
2.2.1
System:
Mac OS X
bitness:
64-bit
hardware architecture:
amd64/x86

Description

The macOS app is missing the file source:data/kernels/common.h which is supposed to be located at darktable.app/Contents/Resources/share/darktable/kernels/common.h, together with all the other OpenCL kernels. Therefore, some OpenCL kernels fail to build and OpenCL is globally deactivated in darktable.

Solution: Put source:data/kernels/common.h into darktable.app/Contents/Resources/share/darktable/kernels/common.h

I was able to fix OpenCL support locally by putting the file there. But you should fix your build system as well.

darktable_opencl_1st_run.log (14.6 KB) darktable_opencl_1st_run.log common.h not found when compiling nlmeans.cl Christian Stussak, 01/04/2017 11:05 AM
darktable_opencl_2nd_run.log (37 KB) darktable_opencl_2nd_run.log common.h not found when compiling basecurve.cl Christian Stussak, 01/04/2017 11:05 AM
darktable_opencl_3rd_run.log (104 KB) darktable_opencl_3rd_run.log all kernels compiled successfully Christian Stussak, 01/04/2017 11:05 AM

History

#1 Updated by Roman Lebedev over 3 years ago

  • Assignee set to Igor Kuzmin
  • Subject changed from OpenCL support broken due to missing common.h to OSX DMG: OpenCL support broken due to missing common.h

#2 Updated by Igor Kuzmin over 3 years ago

  • % Done changed from 0 to 20
  • Status changed from New to Incomplete

I have no idea what are you talking about, "make install" installs common.h in correct path and DMG contains it also. OpenCL works for me.
How exactly did you manage to come up with darktable.app without that file?

#3 Updated by Christian Stussak over 3 years ago

I thought this issue would be trivial to track down and resolve, but I was wrong. Also, my initial analysis was too brief and wrong. It doesn't seem to be related to the build system. I downloaded https://github.com/darktable-org/darktable/releases/download/release-2.2.1/darktable-2.2.1.3.dmg again and the file common.h is there like it should.

Actually, I never checked if it was ever missing before. darktable complained that it could not find the file. I copied it over (via command line) and restarted darktable. OpenCL was working again. After your comment, I was about to mark this issue as invalid.

However, this issue was a mystery to me (I also felt kind of stupid ...) and digged deeper into it.

After rm -r ~/.cache/darktable/cached_kernels_for_*, I did /Applications/darktable.app/Contents/MacOS/darktable -d opencl and voilà, it complained about the missing common.h again.

I did this a couple of times: delete the OpenCL cache and restart darktable. The affected kernels sometimes where the same, sometimes they changed. After a couple of runs of darktable without deleting the cache, darktable finally succeeded to compile and cache all of the OpenCL kernels. Then I deleted the cache, started dt again: it worked on the first run. Next round: It took 6 attempts to compile all shaders correctly.

Since the 'common.h' not found issue is triggered non-deterministic, my impression is that this is related to a race condition. Maybe something related to file locking of common.h? (just guessing)

I attached the logs for three consecutive runs of /Applications/darktable.app/Contents/MacOS/darktable -d opencl. I deleted the cache only before the first run. Here are the essential parts:

1st run:

[opencl_init] compiling program `nlmeans.cl' ..
[opencl_fopen_stat] could not open file `/Users/stussak/.cache/darktable/cached_kernels_for_HDGraphics4000/nlmeans.cl.bin'!
[opencl_load_program] could not load cached binary program, trying to compile source
[opencl_load_program] successfully loaded program from `/Applications/darktable.app/Contents/Resources/share/darktable/kernels/nlmeans.cl'
[opencl_build_program] could not build program: -11
[opencl_build_program] BUILD STATUS: -2
BUILD LOG:
<program source>:20:10: fatal error: 'common.h' file not found
#include "common.h" 
         ^

[opencl_init] failed to compile program `nlmeans.cl'!
[opencl_init] FINALLY: opencl is NOT AVAILABLE on this system.
[opencl_init] initial status of opencl enabled flag is OFF.

2nd run:

[opencl_init] compiling program `basecurve.cl' ..
[opencl_fopen_stat] could not open file `/Users/stussak/.cache/darktable/cached_kernels_for_HDGraphics4000/basecurve.cl.bin'!
[opencl_load_program] could not load cached binary program, trying to compile source
[opencl_load_program] successfully loaded program from `/Applications/darktable.app/Contents/Resources/share/darktable/kernels/basecurve.cl'
[opencl_build_program] could not build program: -11
[opencl_build_program] BUILD STATUS: -2
BUILD LOG:
<program source>:20:10: fatal error: 'common.h' file not found
#include "common.h" 
         ^

[opencl_init] failed to compile program `basecurve.cl'!
[opencl_init] FINALLY: opencl is NOT AVAILABLE on this system.
[opencl_init] initial status of opencl enabled flag is OFF.

3rd run:

...
compiling kernels
...
[opencl_init] kernel loading time: 5.3044
[opencl_init] OpenCL successfully initialized.
[opencl_init] here are the internal numbers and names of OpenCL devices available to darktable:
[opencl_init]        0    'HD Graphics 4000'
[opencl_init]        1    'GeForce GT 650M'
[opencl_init] these are your device priorities:
[opencl_init]         image    preview    export    thumbnail
[opencl_init]        0    1    0    0
[opencl_init]        1    -1    1    1
[opencl_init] FINALLY: opencl is AVAILABLE on this system.
[opencl_init] initial status of opencl enabled flag is ON.

#4 Updated by Christian Stussak over 3 years ago

While the above log files suggest that only HD Graphics 4000 is affected, I was now able to reproduce with GeForce GT 650M as well. So it doesn't seem to be vendor related.

#5 Updated by Igor Kuzmin over 3 years ago

  • % Done changed from 20 to 10
  • Assignee deleted (Igor Kuzmin)
  • Status changed from Incomplete to Confirmed
  • Category changed from Buildsystem to General
  • Subject changed from OSX DMG: OpenCL support broken due to missing common.h to Mac: OpenCL kernels compilation sporadically complain about missing common.h

Confirmed, but I have no idea why it's happening (or did it actually start with 2.2 release). Include dir is specified as compiler option (-I) for every kernel, I can't see any race condition. I would consider that as one of the many OpenCL issues on Mac. I've left the issue opened, but removed myself from assignees, because I won't investigate further.

#6 Updated by Christian Stussak over 3 years ago

I can't tell wether it started with 2.2.0. Probably not. But during the past days, I was digging into a couple of issues I experienced with darktable recently (I finally had time for that). Usually, I just start darktable through the menu, but for the investigations, I started darktable from the command line (-d all) and spotted the OpenCL problems.

I just checked (via fs_usage) which files are accessed by the OpenCL compiler. I expected to see an error related to access restrictions or file not found for .../share/darktable/kernels/common.h. Instead, I got this surprising result: During the run where the compilation fails, the OpenCL compiler does not even attempt to access the correct file. It just ignores the path given by the -I option (which is correctly given, I checked this as well). Instead, it tries to open common.h in various system path know to the OpenCL compiler and, of course, can't find anything (ENOENT). It is using the correct .../share/darktable/kernels/common.h path for all other runs.

So, this looks like an issue with the OpenCL compiler on macOS (I am on 10.11.6).

Luckily, we have a workaround: Restart darktable a couple of times. It will eventually succeed compiling all OpenCL kernels. Due to the cache for compiled kernels, you only need to do this after each program update. May this serve someone who runs into the same issue, I ran into.

#7 Updated by Jilles van Gurp over 1 year ago

I'm running into this with a recent macbook pro with a AMD Radeon Pro 560 on both 2.4.4 and 2.6.0Rc2. This is still an issue and I can confirm the workaround of starting multiple times still works. Thanks Christian!

After reading this thread, I verified the problem with ./darktable-cltest. It indeed complains about common.h

My imac 5k used to have the same issue and started working after mojave 10.14.1 a few weeks ago. I first blamed this on a driver issue that apparently got fixed but after reading this thread, I decided to give restarting darktable a few times a try on my mac book.

A few observations that may be helpful:
- the number of files in .cache grows between restarts. It is definitely succeeding at building progressively more. This means there definitely is some issue with the build.
- if fails for both the intel 630 and amd 560 initially. Judging from the file names, it actually gives up before even trying to build the files for the 560 the first time.
- at some point the compilation succeeds and I can turn on opencl

I hope this helps.

Also available in: Atom PDF

Go to top