Project

General

Profile

Camera Support #11980

Panasonic DMC-ZS100 noise profile

Added by David Gasaway 8 months ago. Updated 5 months ago.

Status:
Closed: invalid
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
01/30/2018
Due date:
% Done:

0%

Affected Version:
2.4.0

Description

I'm working on getting noise profile data for the DMC-ZS100. Currently, it seems the gen-profile.sh script is not working because my darktable-cli is unable to decode the raw files (#11950). Is there a way to get around this for now (say, use the DMC-TZ100 raw support)?

dt-noiseprofile-20180131.tar.gz (2.58 MB) David Gasaway, 02/01/2018 05:43 AM

dt-noiseprofile-20180201.tar.gz (3.25 MB) David Gasaway, 02/02/2018 06:48 AM

History

#1 Updated by David Gasaway 8 months ago

I managed to get past that by building the git master source and a little hacking on the noise scripts (path to darktable-cli and #11963). However, gen-profile seemed to stop on the first file, and the only PDF is zero bytes.

===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/dave/Projects/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dave/Projects/darktable/tools/noise'
--> Check for tethering tools availability

NOTE: Tethering tools are missing; you'll need to provide input images
yourself.
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 125 image: ./P1010077.RW2
--> Found ISO 200 image: ./P1010078.RW2
--> Found ISO 400 image: ./P1010079.RW2
--> Found ISO 800 image: ./P1010080.RW2
--> Found ISO 1600 image: ./P1010081.RW2
--> Found ISO 3200 image: ./P1010082.RW2
--> Found ISO 6400 image: ./P1010083.RW2
--> Found ISO 12800 image: ./P1010084.RW2

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 125:
    ./P1010077.RW2
--> ISO 200:
    ./P1010078.RW2
--> ISO 400:
    ./P1010079.RW2
--> ISO 800:
    ./P1010080.RW2
--> ISO 1600:
    ./P1010081.RW2
--> ISO 3200:
    ./P1010082.RW2
--> ISO 6400:
    ./P1010083.RW2
--> ISO 12800:
    ./P1010084.RW2

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.

===> Profile image for "Panasonic - DMC-ZS100 - 125 ISO" 
--> Converting ./P1010077.RW2 (ISO 125)
RW2 IMAGE
RW2 IMAGE
[export_job] exported to `./P1010077.pfm'
--> Run noiseprofile
--> Plotting ./P1010077.pfm

#2 Updated by Stefan Schöfegger 8 months ago

_--> Plotting ./P1010077.pfm is the last line?
It should look like

===> Profile image for "Sony - NEX-6 - 1600 ISO" 
--> Converting ./DSC06767.ARW (ISO 1600)
[export_job] exported to `./DSC06767.pfm'
--> Run noiseprofile
--> Plotting ./DSC06767.pfm
--> Fitted parametric curves
--> Flattened ./DSC06767.pfm
--> Save generated preset

===> Profile image for "Sony - NEX-6 - 1600 ISO" 
--> Converting ./DSC06768.ARW (ISO 1600)
[export_job] exported to `./DSC06768.pfm'
--> Run noiseprofile
--> Plotting ./DSC06768.pfm
--> Fitted parametric curves
--> Flattened ./DSC06768.pfm
--> Save generated preset

Maybe something wrong with gnuplot?
What does
gnuplot --version

give? Are there any *.dat files in the image directory? Can you debug starting in gen-profile line 145?

*.dat should look something like

0.000000 0.000450 0.000392 0.000401 1632833.000000 803477.000000 2615774.000000 0.000000 0.000000 0.000000
0.003333 0.000623 0.000496 0.000738 1191055.000000 1258868.000000 992342.000000 0.000538 0.000366 0.000699
0.006667 0.000772 0.000594 0.000873 660678.000000 642407.000000 337906.000000 0.001282 0.000830 0.001985
0.010000 0.000868 0.000674 0.001063 396341.000000 436548.000000 32162.000000 0.002206 0.001385 0.003509
0.013333 0.000974 0.000744 0.001156 91769.000000 384683.000000 5588.000000 0.003244 0.002015 0.005363
...

#3 Updated by David Gasaway 8 months ago

Stefan Schöfegger wrote:

_--> Plotting ./P1010077.pfm is the last line?

Yes, it would return to the prompt after that output. What I pasted is the entire STDOUT from the run.

Maybe something wrong with gnuplot?

[dave@chopin noise]$ gnuplot --version
gnuplot 5.2 patchlevel 2 (Gentoo revision r0)

Are there any *.dat files in the image directory?

[dave@chopin noise]$ ls -l *.dat
-rw------- 1 dave dave 29424 Jan 29 22:52 P1010077.dat

Can you debug starting in gen-profile line 145?

I'm no expert on debugging shell scripts. How do you mean?

*.dat should look something like

Yes, it has 300 lines that look like that.

Thanks.

#4 Updated by David Gasaway 8 months ago

It occurs to me that, since I use funtoo (gentoo derivative), there are a number of USE flags that are not enabled for gnuplot. Would it be needing any of the following gnuplot features which are disabled?
-aqua -bitmap -cairo -compat -doc -examples -ggi -latex -libcaca -libcerf -lua -regis -svga -wxwidgets

#5 Updated by David Gasaway 8 months ago

I needed to build gnuplot with cairo (consider adding to the instructions).

Still, before I upload, I'd like clarification. The procedure says to apply the 'raw linear' style (no mention of export in this paragraph). Later it says 'go to the directory you just exported your images to'. So, I'm supposed to export the images to some non-raw format? I don't know how to export back to a raw format like 'RW2'. Yet, your example shows 'ARW' which is a Sony raw format, if I'm not mistaken. Is the procedure just trying to say that I need to apply the style so that there is an XMP file with the correct history stack?

#6 Updated by Tobias Ellinghaus 8 months ago

These days you don't need to do anything to the raw files at all, not even import them into darktable. The script does all that on its own using darktable-cli.

#7 Updated by David Gasaway 8 months ago

Cool. Here are the results. I'm having trouble using the presets.json file though. When I run 'darktable --noiseprofiles ./presets.json' and open the denoise (profiled) module, the only profile available is generic poissonian. The entries from the json file aren't there.

#8 Updated by Stefan Schöfegger 8 months ago

See https://redmine.darktable.org/issues/11977#note-2 for explanation why you don't see your entries. For ZS100 you have to use TZ100. I'll change it local.
I miss an entry for 25600 which is availiable only when [Extended ISO] is [ON]. Please run the script again with iso 25600 image.

#9 Updated by David Gasaway 8 months ago

You don't want all the extended ISOs, just 25600?

#10 Updated by David Gasaway 8 months ago

New results with ISO 25600. Seems to be leaving a lot of red noise with ISO 6400 and up. Is that to be expected?

#11 Updated by David Gasaway 7 months ago

Are my images adequate?

#12 Updated by Stefan Schöfegger 7 months ago

  • Status changed from New to Triaged
  • % Done changed from 0 to 20

Your result looks good. I missed that for DMC-TZ100 there is already a noise profile included (which is also used for DMC-ZS100). Do you have any problems with it?

#13 Updated by David Gasaway 7 months ago

Stefan Schöfegger wrote:

Your result looks good. I missed that for DMC-TZ100 there is already a noise profile included (which is also used for DMC-ZS100). Do you have any problems with it?

Until the ZS100 raw support was there, I didn't have a way to test it. And discussion on issue #11950 lead me to believe that I should submit a ZS100 specific noise profile. Not true? I have a local build that I could play with.

#14 Updated by Stefan Schöfegger 6 months ago

  • Status changed from Triaged to Closed: invalid
  • % Done changed from 20 to 0

#15 Updated by David Gasaway 6 months ago

Sorry, I guess I never did get back to you. I was getting identical results whether I used the --noiseprofiles argument or not, so I wasn't sure if I was testing it right.

#16 Updated by Roman Lebedev 5 months ago

  • Target version set to 2.6.0

Also available in: Atom PDF