Project

General

Profile

Camera Support #11539

Canon 5D Mark IV noise profiles

Added by Dejan Perkovic 9 months ago. Updated 6 months ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
03/14/2017
Due date:
% Done:

100%

Affected Version:
2.2.3

Description

Hi,

I have trouble running gen-profile resulting with "Incorrect green channel" error. My darktable is the latest Ubuntu version (2.2.3) and the tools directory was downloaded and compiled today (also 2.2.3). Before running the gen-profile, I used darktable to apply the raw linear style from the tools/noise and generate .XMP (no need to export or anything else, I believe). The resulting image looks greenish, but jpegs gen-profile creates look normal (like it has not used .XMP)

The log for my failing sample picture is below. Raw and result files are attached. Any hints?

BTW, congrats on excellent software.

dejanp@leopardx:~/Downloads/darktable-release-2.2.3/tools/noise/5div$ ../gen-profile -i 400 -d ~/Pictures/noiseprofile
===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 400 image: /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 400:
/home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.

===> Profile image for "Canon - EOS 5D Mark IV - 400 ISO"
--> Converting /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2 (ISO 400)
[export_job] exported to `/home/dejanp/Pictures/noiseprofile/7M9A0061.pfm'
--> Run noiseprofile
--> Plotting /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Fitted parametric curves
--> Flattened /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Save generated preset

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

7M9A0061-large.jpg (13.9 MB) Dejan Perkovic, 03/14/2017 08:05 AM

7M9A0061.pdf (26.2 KB) Dejan Perkovic, 03/14/2017 08:05 AM

7M9A0061_flat.pdf (17.3 KB) Dejan Perkovic, 03/14/2017 08:06 AM

7M9A0061.CR2 (31.1 MB) Dejan Perkovic, 03/14/2017 08:06 AM

7M9A0061.CR2.xmp (4.24 KB) Dejan Perkovic, 03/14/2017 08:17 AM

noise_result.pdf - for ISO 3200 (99.5 KB) Dejan Perkovic, 03/14/2017 11:48 AM

dt-noiseprofile-20170319.tar.gz (11.7 MB) Dejan Perkovic, 03/20/2017 08:40 AM

presets.json (6.11 KB) Dejan Perkovic, 03/20/2017 08:41 AM

benchmark.output (748 Bytes) Dejan Perkovic, 03/20/2017 08:41 AM

dt-noiseprofile-20170612.tar.gz (9.4 MB) David Houlder, 06/12/2017 03:54 PM

dh-vs-11539.gnuplot (8.58 KB) David Houlder, 06/12/2017 03:55 PM

Associated revisions

Revision e48c1684
Added by David Houlder 6 months ago

noise profile: Add Canon EOS 5D Mark IV, fixes #11539

Profile generated by Dejan Perkovic. See
https://redmine.darktable.org/issues/11539#note-4

(cherry picked from commit 492b7d44b82b7db0dfff73edc7d99f6c5e9253af)

Fixes #1507

Thanks to Stefan Schöfegger for review!

History

#1 Updated by Roman Lebedev 9 months ago

Dejan Perkovic wrote:

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

Looking at the histogram graphs, you can see that the curve labelled "green channel" starts from ~0.01, not from 0, like e.g. blue channel does
Meaning, there are no completely black pixels. So you should play with less exposure (shorter shutter / narrower aperture).
But do note that the image must have both the fully overexposed pixels, and fully underexposed pixels...

#2 Updated by Dejan Perkovic 9 months ago

Thanks. I succeeded getting the profiles with just a bit less exposure and some added dark elements in the shade. Now I have a different challenge in that when testing the generated profile against the source picture in darktable with darktable --noiseprofiles, it appears that NR does worse then no NR, and I mean by just comparing noise. I did a test with ISO 400 where there is not that much noise to begin with, but with ISO 3200, I see that profiled NR does little at all while non-profiled denoising modules do a lot NR reduction. Perhaps my source images are still deficient.

Roman Lebedev wrote:

Dejan Perkovic wrote:

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

Looking at the histogram graphs, you can see that the curve labelled "green channel" starts from ~0.01, not from 0, like e.g. blue channel does
Meaning, there are no completely black pixels. So you should play with less exposure (shorter shutter / narrower aperture).
But do note that the image must have both the fully overexposed pixels, and fully underexposed pixels...

#3 Updated by Dejan Perkovic 9 months ago

So, I did go through all the ISOs w/o errors from gen-profile, but the fit doesn't seem to be there (similar to the linked PDF in previous message). I tried using it and it works, but I wouldn't call it simpler or better then denoise-non-local w/o profile. I have been taking bracketed shots with different exposures and the results would be very different, too (see below). I don't know if that is ok (no idea about the meaning behind the numbers) nor how these numbers are taken (averaged?) by darktable. Any hints on how I should proceed? I have the photos (light fixture in a dark room) and all the PDFs...

{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [3.5456631955808e-06, 1.34515466146825e-06, 9.732281960835e-06], "b": [5.28539268622165e-07, 4.37625906597106e-07, 4.83807080438081e-07]},
{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [3.21454586190599e-06, 1.42182437654684e-06, 9.88120015198096e-06], "b": [5.682355293372e-07, 4.91537986243456e-07, 5.48884695137732e-07]},
{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [5.3267690912655e-06, 1.36428574398318e-06, 1.00242868077735e-05], "b": [4.01123445962802e-07, 4.00631579426288e-07, 4.21621188290524e-07]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [0.000137537183586954, 6.21441077049712e-05, 0.000380640374226419], "b": [1.76936825097717e-05, 1.38344898916574e-05, 9.34108671384824e-06]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [0.000210720832716183, 6.96852744171812e-05, 0.000363800625511486], "b": [1.30983771101841e-05, 1.20756324969068e-05, 1.05305147675367e-05]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [8.90252994876857e-05, 3.1284746884955e-05, 0.000328537048468565], "b": [2.19753216911513e-05, 1.88880620335612e-05, 1.5172919745239e-05]},

Dejan Perkovic wrote:

Thanks. I succeeded getting the profiles with just a bit less exposure and some added dark elements in the shade. Now I have a different challenge in that when testing the generated profile against the source picture in darktable with darktable --noiseprofiles, it appears that NR does worse then no NR, and I mean by just comparing noise. I did a test with ISO 400 where there is not that much noise to begin with, but with ISO 3200, I see that profiled NR does little at all while non-profiled denoising modules do a lot NR reduction. Perhaps my source images are still deficient.

#4 Updated by Dejan Perkovic 9 months ago

I made it working in the end (guessing the export defaults in darktable being a problem). Attached are the tar file and presets.json. Below is the result of benchmark (PSNR of original vs denoise moudle with default parameters). Interesting to see PSNRs of non full stop ISOs.

Let me know if you need anything else,
Dejan
--

iso50 : 20.7386 20.7498
iso100 : 42.7686 45.4188
iso125 : 41.5104 44.4125
iso160 : 29.2676 29.4696
iso200 : 39.4004 41.7618
iso250 : 38.5898 41.3635
iso320 : 24.9604 25.1441
iso400 : 35.8072 38.4956
iso500 : 34.6316 37.4097
iso640 : 24.7774 25.1649
iso800 : 33.7659 38.0497
iso1000 : 32.9191 37.8598
iso1250 : 24.3282 25.1178
iso1600 : 31.2177 37.0997
iso2000 : 30.0234 35.704
iso2500 : 23.6142 25.1936
iso3200 : 28.2885 35.0616
iso4000 : 27.3969 34.801
iso5000 : 22.3141 25.0617
iso6400 : 25.5176 33.0572
iso8000 : 24.4749 32.1006
iso10000 : 20.1266 24.1101
iso12800 : 22.4491 30.2177
iso16000 : 21.4559 28.8545
iso20000 : 17.6819 22.9577
iso25600 : 19.4315 26.5703
iso32000 : 18.3591 25.575
iso51200 : 16.1277 23.1148
iso102400 : 17.2095 22.4031

Dejan Perkovic wrote:

Hi,

I have trouble running gen-profile resulting with "Incorrect green channel" error. My darktable is the latest Ubuntu version (2.2.3) and the tools directory was downloaded and compiled today (also 2.2.3). Before running the gen-profile, I used darktable to apply the raw linear style from the tools/noise and generate .XMP (no need to export or anything else, I believe). The resulting image looks greenish, but jpegs gen-profile creates look normal (like it has not used .XMP)

The log for my failing sample picture is below. Raw and result files are attached. Any hints?

BTW, congrats on excellent software.

dejanp@leopardx:~/Downloads/darktable-release-2.2.3/tools/noise/5div$ ../gen-profile -i 400 -d ~/Pictures/noiseprofile
===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 400 image: /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 400:
/home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.

===> Profile image for "Canon - EOS 5D Mark IV - 400 ISO"
--> Converting /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2 (ISO 400)
[export_job] exported to `/home/dejanp/Pictures/noiseprofile/7M9A0061.pfm'
--> Run noiseprofile
--> Plotting /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Fitted parametric curves
--> Flattened /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Save generated preset

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

#5 Updated by David Houlder 6 months ago

Hi Dejan...
In the interests of cross-validation, I generated my own noise profile for the 5D Mark IV and got pretty similar results to your final profile. See attached dt-noiseprofile-20170612.tar.gz

Out of curiosity I created a gnuplot script that lets you compare my results against yours in an interactive 3D plot ("a" and "b" coefficients vs. ISO). The solid lines are my fit, the dashed lines are yours, and there are vectors at each ISO showing the discrepancy. If you spin the plot around you'll find that (due to the log axes) the points all line up reasonably well. Run it with...

gnuplot dh-vs-11539.gnuplot

I'll create a pull request soon unless someone advises otherwise.
cheers
David

#6 Updated by David Houlder 6 months ago

Hi Dejan: I just created pull request #1507 using your noise profile from comment #note-4 and attributed it to you. Thanks for your hard work.

#7 Updated by Stefan Schöfegger 6 months ago

The noise profile dt-noiseprofile-20170319.tar.gz does not look optimal. All iso but 1250-8000 show artefacts I had not seen so far, but the errors are small. Normally I would suggest to make new shots. As we have a second noise profile with exactly the same errors ( dt-noiseprofile-20170612.tar.gz ) it seems to be chip specific.
PR 1507 can be merged.

#8 Updated by Roman Lebedev 6 months ago

Hm, were those noiseprofiling shots taken in dual-raw mode?
I wonder if there is a difference.

#9 Updated by David Houlder 6 months ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

#10 Updated by David Houlder 6 months ago

Roman Lebedev wrote:

Hm, were those noiseprofiling shots taken in dual-raw mode?

In my case at least, no. dt-noiseprofile-20170612.tar.gz was generated with dual-pixel mode disabled.

#11 Updated by Roman Lebedev 6 months ago

  • Target version set to 2.4.0

Also available in: Atom PDF