Project

General

Profile

Camera Support #11539

Canon 5D Mark IV noise profiles

Added by Dejan Perkovic about 2 months ago. Updated about 1 month ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
03/14/2017
Due date:
% Done:

0%

Affected Version:
2.2.3

Description

Hi,

I have trouble running gen-profile resulting with "Incorrect green channel" error. My darktable is the latest Ubuntu version (2.2.3) and the tools directory was downloaded and compiled today (also 2.2.3). Before running the gen-profile, I used darktable to apply the raw linear style from the tools/noise and generate .XMP (no need to export or anything else, I believe). The resulting image looks greenish, but jpegs gen-profile creates look normal (like it has not used .XMP)

The log for my failing sample picture is below. Raw and result files are attached. Any hints?

BTW, congrats on excellent software.

dejanp@leopardx:~/Downloads/darktable-release-2.2.3/tools/noise/5div$ ../gen-profile -i 400 -d ~/Pictures/noiseprofile
===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 400 image: /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 400:
/home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.

===> Profile image for "Canon - EOS 5D Mark IV - 400 ISO"
--> Converting /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2 (ISO 400)
[export_job] exported to `/home/dejanp/Pictures/noiseprofile/7M9A0061.pfm'
--> Run noiseprofile
--> Plotting /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Fitted parametric curves
--> Flattened /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Save generated preset

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

7M9A0061-large.jpg (13.9 MB) Dejan Perkovic, 03/14/2017 08:05 AM

7M9A0061.pdf (26.2 KB) Dejan Perkovic, 03/14/2017 08:05 AM

7M9A0061_flat.pdf (17.3 KB) Dejan Perkovic, 03/14/2017 08:06 AM

7M9A0061.CR2 (31.1 MB) Dejan Perkovic, 03/14/2017 08:06 AM

7M9A0061.CR2.xmp (4.24 KB) Dejan Perkovic, 03/14/2017 08:17 AM

noise_result.pdf - for ISO 3200 (99.5 KB) Dejan Perkovic, 03/14/2017 11:48 AM

dt-noiseprofile-20170319.tar.gz (11.7 MB) Dejan Perkovic, 03/20/2017 08:40 AM

presets.json (6.11 KB) Dejan Perkovic, 03/20/2017 08:41 AM

benchmark.output (748 Bytes) Dejan Perkovic, 03/20/2017 08:41 AM

History

#1 Updated by Roman Lebedev about 2 months ago

Dejan Perkovic wrote:

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

Looking at the histogram graphs, you can see that the curve labelled "green channel" starts from ~0.01, not from 0, like e.g. blue channel does
Meaning, there are no completely black pixels. So you should play with less exposure (shorter shutter / narrower aperture).
But do note that the image must have both the fully overexposed pixels, and fully underexposed pixels...

#2 Updated by Dejan Perkovic about 2 months ago

Thanks. I succeeded getting the profiles with just a bit less exposure and some added dark elements in the shade. Now I have a different challenge in that when testing the generated profile against the source picture in darktable with darktable --noiseprofiles, it appears that NR does worse then no NR, and I mean by just comparing noise. I did a test with ISO 400 where there is not that much noise to begin with, but with ISO 3200, I see that profiled NR does little at all while non-profiled denoising modules do a lot NR reduction. Perhaps my source images are still deficient.

Roman Lebedev wrote:

Dejan Perkovic wrote:

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

Looking at the histogram graphs, you can see that the curve labelled "green channel" starts from ~0.01, not from 0, like e.g. blue channel does
Meaning, there are no completely black pixels. So you should play with less exposure (shorter shutter / narrower aperture).
But do note that the image must have both the fully overexposed pixels, and fully underexposed pixels...

#3 Updated by Dejan Perkovic about 2 months ago

So, I did go through all the ISOs w/o errors from gen-profile, but the fit doesn't seem to be there (similar to the linked PDF in previous message). I tried using it and it works, but I wouldn't call it simpler or better then denoise-non-local w/o profile. I have been taking bracketed shots with different exposures and the results would be very different, too (see below). I don't know if that is ok (no idea about the meaning behind the numbers) nor how these numbers are taken (averaged?) by darktable. Any hints on how I should proceed? I have the photos (light fixture in a dark room) and all the PDFs...

{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [3.5456631955808e-06, 1.34515466146825e-06, 9.732281960835e-06], "b": [5.28539268622165e-07, 4.37625906597106e-07, 4.83807080438081e-07]},
{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [3.21454586190599e-06, 1.42182437654684e-06, 9.88120015198096e-06], "b": [5.682355293372e-07, 4.91537986243456e-07, 5.48884695137732e-07]},
{"name": "EOS 5D Mark IV iso 100", "iso": 100, "a": [5.3267690912655e-06, 1.36428574398318e-06, 1.00242868077735e-05], "b": [4.01123445962802e-07, 4.00631579426288e-07, 4.21621188290524e-07]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [0.000137537183586954, 6.21441077049712e-05, 0.000380640374226419], "b": [1.76936825097717e-05, 1.38344898916574e-05, 9.34108671384824e-06]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [0.000210720832716183, 6.96852744171812e-05, 0.000363800625511486], "b": [1.30983771101841e-05, 1.20756324969068e-05, 1.05305147675367e-05]},
{"name": "EOS 5D Mark IV iso 3200", "iso": 3200, "a": [8.90252994876857e-05, 3.1284746884955e-05, 0.000328537048468565], "b": [2.19753216911513e-05, 1.88880620335612e-05, 1.5172919745239e-05]},

Dejan Perkovic wrote:

Thanks. I succeeded getting the profiles with just a bit less exposure and some added dark elements in the shade. Now I have a different challenge in that when testing the generated profile against the source picture in darktable with darktable --noiseprofiles, it appears that NR does worse then no NR, and I mean by just comparing noise. I did a test with ISO 400 where there is not that much noise to begin with, but with ISO 3200, I see that profiled NR does little at all while non-profiled denoising modules do a lot NR reduction. Perhaps my source images are still deficient.

#4 Updated by Dejan Perkovic about 1 month ago

I made it working in the end (guessing the export defaults in darktable being a problem). Attached are the tar file and presets.json. Below is the result of benchmark (PSNR of original vs denoise moudle with default parameters). Interesting to see PSNRs of non full stop ISOs.

Let me know if you need anything else,
Dejan
--

iso50 : 20.7386 20.7498
iso100 : 42.7686 45.4188
iso125 : 41.5104 44.4125
iso160 : 29.2676 29.4696
iso200 : 39.4004 41.7618
iso250 : 38.5898 41.3635
iso320 : 24.9604 25.1441
iso400 : 35.8072 38.4956
iso500 : 34.6316 37.4097
iso640 : 24.7774 25.1649
iso800 : 33.7659 38.0497
iso1000 : 32.9191 37.8598
iso1250 : 24.3282 25.1178
iso1600 : 31.2177 37.0997
iso2000 : 30.0234 35.704
iso2500 : 23.6142 25.1936
iso3200 : 28.2885 35.0616
iso4000 : 27.3969 34.801
iso5000 : 22.3141 25.0617
iso6400 : 25.5176 33.0572
iso8000 : 24.4749 32.1006
iso10000 : 20.1266 24.1101
iso12800 : 22.4491 30.2177
iso16000 : 21.4559 28.8545
iso20000 : 17.6819 22.9577
iso25600 : 19.4315 26.5703
iso32000 : 18.3591 25.575
iso51200 : 16.1277 23.1148
iso102400 : 17.2095 22.4031

Dejan Perkovic wrote:

Hi,

I have trouble running gen-profile resulting with "Incorrect green channel" error. My darktable is the latest Ubuntu version (2.2.3) and the tools directory was downloaded and compiled today (also 2.2.3). Before running the gen-profile, I used darktable to apply the raw linear style from the tools/noise and generate .XMP (no need to export or anything else, I believe). The resulting image looks greenish, but jpegs gen-profile creates look normal (like it has not used .XMP)

The log for my failing sample picture is below. Raw and result files are attached. Any hints?

BTW, congrats on excellent software.

dejanp@leopardx:~/Downloads/darktable-release-2.2.3/tools/noise/5div$ ../gen-profile -i 400 -d ~/Pictures/noiseprofile
===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dejanp/Downloads/darktable-release-2.2.3/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 400 image: /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 400:
/home/dejanp/Pictures/noiseprofile/7M9A0061.CR2

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.

===> Profile image for "Canon - EOS 5D Mark IV - 400 ISO"
--> Converting /home/dejanp/Pictures/noiseprofile/7M9A0061.CR2 (ISO 400)
[export_job] exported to `/home/dejanp/Pictures/noiseprofile/7M9A0061.pfm'
--> Run noiseprofile
--> Plotting /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Fitted parametric curves
--> Flattened /home/dejanp/Pictures/noiseprofile/7M9A0061.pfm
--> Save generated preset

$\033[31mERROR: Incorrect green channel.

Possible reason:
o The input RAW image is bad regarding lighting or exposure.
o You may have set a default output profile in darktable which is
unsuitable for noise profiling.

If the latter reason applies to you, change it back to "image settings"
and re-run this script.$\033[0m

Also available in: Atom PDF