Project

General

Profile

Bug #11574

Failed to create a noise profile with tools/noise/gen-profile — it detects input raw images (Canon G16), but the output is incomplete and the actual noise profile is missing

Added by Sarge Borsch about 2 years ago. Updated about 2 years ago.

Status:
Closed: invalid
Priority:
Low
Assignee:
-
Category:
General
Target version:
Start date:
04/15/2017
Due date:
% Done:

0%

Affected Version:
git master branch
System:
Mac OS X
bitness:
64-bit
hardware architecture:
amd64/x86

Description

I tried to create a noise profile from raw images taken as described in https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/
for Canon G16.

Log is in attached file.
File list after running the script:

$ ls
IMG_0154-large.jpg
IMG_0154-thumb.jpg
IMG_0154.CR2
IMG_0154.dat
IMG_0154.fit
IMG_0154.pdf
IMG_0154.pfm
IMG_0155-large.jpg
IMG_0155-thumb.jpg
IMG_0155.CR2
IMG_0156-large.jpg
IMG_0156-thumb.jpg
IMG_0156.CR2
IMG_0157-large.jpg
IMG_0157-thumb.jpg
IMG_0157.CR2
IMG_0158-large.jpg
IMG_0158-thumb.jpg
IMG_0158.CR2
IMG_0159-large.jpg
IMG_0159-thumb.jpg
IMG_0159.CR2
IMG_0160-large.jpg
IMG_0160-thumb.jpg
IMG_0160.CR2
IMG_0161-large.jpg
IMG_0161-thumb.jpg
IMG_0161.CR2
IMG_0162-large.jpg
IMG_0162-thumb.jpg
IMG_0162.CR2
output-files-list.txt

Noise profile is missing here, also only the first image got converted to PFM.
Also, not sure if this is as expected, but JPG files don't look right — they seem to NOT have "raw linear" style used.

I've uploaded test raw images:
https://my.mixtape.moe/appcbh.tbz2 — part 1
https://my.mixtape.moe/yywzft.tbz2 — part 2
https://my.mixtape.moe/easbue.tbz2 — part 3
Maybe someone else could create a noise profile from them.

log.txt Magnifier - Full log of `tools/noise/gen-profile -d . &> log.txt` (2.14 KB) Sarge Borsch, 04/15/2017 08:33 AM

plots.tar.gz - plots generated by gen-profile (251 KB) Sarge Borsch, 04/15/2017 10:15 AM

IMG_0154.png (602 KB) Sarge Borsch, 04/15/2017 10:47 AM

IMG_0155.png (649 KB) Sarge Borsch, 04/15/2017 10:47 AM

IMG_0157.png (897 KB) Sarge Borsch, 04/15/2017 10:47 AM

IMG_0158.png (1.04 MB) Sarge Borsch, 04/15/2017 10:48 AM

IMG_0156.png (775 KB) Sarge Borsch, 04/15/2017 10:48 AM

IMG_0159.png (1.18 MB) Sarge Borsch, 04/15/2017 10:48 AM

IMG_0160.png (1.31 MB) Sarge Borsch, 04/15/2017 10:48 AM

IMG_0161.png (1.45 MB) Sarge Borsch, 04/15/2017 10:48 AM

IMG_0162.png (1.59 MB) Sarge Borsch, 04/15/2017 10:48 AM

History

#1 Updated by Roman Lebedev about 2 years ago

Sarge Borsch wrote:

convert: clBuildProgram failed. (-11) @ warning/opencl.c/CompileOpenCLKernel/1365.

Does dt work normally otherwise?
I strongly suspect you need to disable opencl, either in darktable, or pass --disable-opencl as command-line option

#2 Updated by Sarge Borsch about 2 years ago

Roman Lebedev wrote:

Sarge Borsch wrote:

convert: clBuildProgram failed. (-11) @ warning/opencl.c/CompileOpenCLKernel/1365.

Does dt work normally otherwise?
I strongly suspect you need to disable opencl, either in darktable, or pass --disable-opencl as command-line option

Yes, darktable works normally otherwise. (Not sure if it's working at full speed though, sometimes it's really slow, but most of the times not too slow)
I'll try it without OpenCL

upd.: where should I disable it or put this option? I tried to disable OpenCL in settings and add this option to `gen-profile` but it didn't change anything

#3 Updated by Sarge Borsch about 2 years ago

Also tried to add --disable-opencl to the darktable-cli invokation inside the gen-profile script, but it also didn't change anything, even OpenCL warnings are still there. But I doubt they are the culprit, because first PFM file is created okay.

Maybe it's gnuplot failure? I see that gnuplot errors are silently suppressed in the script.
Maybe it wants a specific version of gnuplot or something? Or it just can't work on systems other than GNU/Linux?

#4 Updated by Sarge Borsch about 2 years ago

I tried deleting the error redirection from the script (which silences gnuplot errors) and this is the hidden output of gnuplot stderr:

gnuplot> set term pdf
                  ^
         line 0: unknown or ambiguous terminal type; type just 'set terminal' for a list

WARNING: Plotting with an 'unknown' terminal.
No output will be generated. Please select a terminal with 'set terminal'.
         line 0: warning: 
    > Implied independent variable y not found in fit function.
    > Assuming version 4 syntax with zerror in column 3 but no zerror keyword.

iter      chisq       delta/lim  lambda   a1            b1           
   0 1.3433643234e+08   0.00e+00  7.49e+02    1.000000e-01   1.000000e-02
   1 1.1832372240e+05  -1.13e+08  7.49e+01    5.713234e-02  -3.266114e-04
   2 2.8261045217e+01  -4.19e+08  7.49e+00    9.099961e-04  -5.866996e-06
   3 9.6253541762e-05  -2.94e+10  7.49e-01    1.665308e-05   2.459610e-08
   4 9.5539188239e-05  -7.48e+02  7.49e-02    1.651103e-05   2.553331e-08
   5 9.5539188239e-05  -1.84e-09  7.49e-03    1.651103e-05   2.553331e-08
iter      chisq       delta/lim  lambda   a1            b1           

After 5 iterations the fit converged.
final sum of squares of residuals : 9.55392e-05
rel. change during last iteration : -1.84409e-14

degrees of freedom    (FIT_NDF)                        : 104
rms of residuals      (FIT_STDFIT) = sqrt(WSSR/ndf)    : 0.00095846
variance of residuals (reduced chisquare) = WSSR/ndf   : 9.18646e-07
p-value of the Chisq distribution (FIT_P)              : 1

Final set of parameters            Asymptotic Standard Error
=======================            ==========================
a1              = 1.6511e-05       +/- 1.611e-07    (0.9757%)
b1              = 2.55333e-08      +/- 1.379e-09    (5.401%)

correlation matrix of the fit parameters:
                a1     b1     
a1              1.000 
b1             -0.768  1.000 
         line 0: warning: 
    > Implied independent variable y not found in fit function.
    > Assuming version 4 syntax with zerror in column 3 but no zerror keyword.

iter      chisq       delta/lim  lambda   a2            b2           
   0 9.6638586404e+07   0.00e+00  4.06e+02    1.000000e-01   1.000000e-02
   1 2.1092137581e+04  -4.58e+08  4.06e+01    1.773394e-02  -1.747910e-04
   2 1.3145766778e-01  -1.60e+10  4.06e+00    5.336693e-05  -4.784076e-07
   3 4.6469290424e-05  -2.83e+08  4.06e-01    8.742861e-06   1.962539e-08
   4 4.6469206473e-05  -1.81e-01  4.06e-02    8.741733e-06   1.963799e-08
iter      chisq       delta/lim  lambda   a2            b2           

After 4 iterations the fit converged.
final sum of squares of residuals : 4.64692e-05
rel. change during last iteration : -1.80659e-06

degrees of freedom    (FIT_NDF)                        : 239
rms of residuals      (FIT_STDFIT) = sqrt(WSSR/ndf)    : 0.000440944
variance of residuals (reduced chisquare) = WSSR/ndf   : 1.94432e-07
p-value of the Chisq distribution (FIT_P)              : 1

Final set of parameters            Asymptotic Standard Error
=======================            ==========================
a2              = 8.74173e-06      +/- 5.428e-08    (0.621%)
b2              = 1.9638e-08       +/- 7.82e-10     (3.982%)

correlation matrix of the fit parameters:
                a2     b2     
a2              1.000 
b2             -0.769  1.000 
         line 0: warning: 
    > Implied independent variable y not found in fit function.
    > Assuming version 4 syntax with zerror in column 3 but no zerror keyword.

iter      chisq       delta/lim  lambda   a3            b3           
   0 1.9530148638e+08   0.00e+00  7.82e+02    1.000000e-01   1.000000e-02
   1 1.2643747916e+05  -1.54e+08  7.82e+01    6.650354e-02  -1.572277e-04
   2 5.6376346481e+01  -2.24e+08  7.82e+00    1.437158e-03  -4.023288e-06
   3 8.9085880191e-04  -6.33e+09  7.82e-01    2.187452e-05   4.760235e-08
   4 8.8819075123e-04  -3.00e+02  7.82e-02    2.156657e-05   4.848839e-08
   5 8.8819075123e-04  -1.44e-09  7.82e-03    2.156657e-05   4.848839e-08
iter      chisq       delta/lim  lambda   a3            b3           

After 5 iterations the fit converged.
final sum of squares of residuals : 0.000888191
rel. change during last iteration : -1.44041e-14

degrees of freedom    (FIT_NDF)                        : 149
rms of residuals      (FIT_STDFIT) = sqrt(WSSR/ndf)    : 0.00244152
variance of residuals (reduced chisquare) = WSSR/ndf   : 5.96101e-06
p-value of the Chisq distribution (FIT_P)              : 1

Final set of parameters            Asymptotic Standard Error
=======================            ==========================
a3              = 2.15666e-05      +/- 4.603e-07    (2.134%)
b3              = 4.84884e-08      +/- 2.232e-09    (4.604%)

correlation matrix of the fit parameters:
                a3     b3     
a3              1.000 
b3             -0.592  1.000 
WARNING: Plotting with an 'unknown' terminal.
No output will be generated. Please select a terminal with 'set terminal'.

So the gnuplot failed to do what it was asked, right?
How can I fix it?

#5 Updated by Roman Lebedev about 2 years ago

I would guess it's again because of macos being so special.
I'd try installing proper gnuplot.

#6 Updated by Sarge Borsch about 2 years ago

Solved it by rebuilding gnuplot with enabled pdf terminal support.

Just to check it's all indeed OK, here's archive with all the generated plots, can you please take a look and decide if they make sense?
I think the overall look is similar to the plot in the article but I'm not an expert in maths…

#7 Updated by Sarge Borsch about 2 years ago

Also, the preset is still not generated, here's the next error:

--> Save generated preset
sort: invalid option -- V
Try `sort --help' for more information.

===> Prepare final tarball
tar: : Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

What can be done here?

#8 Updated by Roman Lebedev about 2 years ago

I don't know for sure, but i'd say they are not overexposed enough.

#9 Updated by Sarge Borsch about 2 years ago

Tried to change sort to gsort (GNU sort) in the script, now it seems to work! ;D

#10 Updated by Sarge Borsch about 2 years ago

Roman Lebedev wrote:

I don't know for sure, but i'd say they are not overexposed enough.

When open them in darktable, apply "raw linear" style, and look at them, the highlight is solid white. Should this area be larger?
Attached thumbnails of the conversion with "raw linear" style (and sRGB output profile), all of the images have clipped highlights.

Also, created a PR: https://github.com/darktable-org/darktable/pull/1471

#11 Updated by Sarge Borsch about 2 years ago

Since this issue is about the gen-profile tool, and it turned out that the only issue here is not-very-obvious error feedback, I think this issue might be closed. But I can't close it myself.

#12 Updated by Tobias Ellinghaus about 2 years ago

  • Status changed from New to Closed: invalid

#13 Updated by Roman Lebedev about 2 years ago

  • Target version set to 2.4.0

Also available in: Atom PDF