Project

General

Profile

Bug #11963

Problem with profiling sensor and photon noise for Panasonic DMC-GF6 camera

Added by Alex Stepanov over 1 year ago. Updated over 1 year ago.

Status:
Closed: upstream
Priority:
Low
Category:
-
Target version:
Start date:
01/23/2018
Due date:
% Done:

0%

Affected Version:
git master branch
System:
other GNU/Linux
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Hello!
I need some help...

I found information how to profile my camera in this place:
https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/
My Panasonic DMC-GF6 camera hasn't a profile for 'denoise (profiled)' module in Darktable.

I could not find the directories and executable files specified in the manual, so I had to download all from branch master https://github.com/darktable-org/darktable.git

First of all, I tried the script on RAW files from the Canon EOD 650d camera and the script worked perfectly! I got a profile, graphs in PDF format and a full set of files. All is OK!

The next step was to do the same for Panasonic DMC-GF6. In accordance with the instruction, I took one shot for each ISO speed with a defocus. After this i run
~/darktable/tools/noise/gen-profile -d .
But, this time, the script does not work until the end and stops with an error.
The problem begins from this line in subr.sh file:
iso=$(get_image_iso "$image")

Full output to console:
@~/darktable/tools/noise/gen-profile -d .

===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution@

I delete all files from the Panasonic camera and replace they (in the same place) by files from the Canon. I started the same script again and it worked successfully.
That is, the problem somehow concerns the RAW files ONLY from the Panasonic DMC-GF6 camera

P1110095.rw2 (18.5 MB) Alex Stepanov, 01/27/2018 04:38 PM

dt-noiseprofile-20180128.tar.gz - Optimal denoise profile for Panasonic DMC-GF6 camera (23 MB) Alex Stepanov, 01/28/2018 04:49 PM

Associated revisions

Revision a5703721
Added by Stefan Schöfegger over 1 year ago

noise profile: Add Panasonic DMC-GF6, refs #11963

Revision 7f573429
Added by Stefan Schöfegger about 1 year ago

noise profile: Add Panasonic DMC-GF6, refs #11963

(cherry picked from commit a5703721842c126e95ac3a3c58ccdbf0730a5cc6)

History

#1 Updated by Alex Stepanov over 1 year ago

One sample image from a Panasonic DMC-GF6 camera that causes an error

#2 Updated by Stefan Schöfegger over 1 year ago

Your image does not trigger an error on my pc. This error is reported a few times now. It is fixed for Nikon in #11580, I have a look into it.

What does

ls -al /bin/sh

output?

Please add one debug message

--- a/tools/noise/gen-profile
+++ b/tools/noise/gen-profile
@@ -65,6 +65,8 @@ fi
 echo "===> Check for required tools" 
 missing_tool=0

+echo "--> Check shell: $SHELL" 
+
 if ! image_info_tools_installed; then
        missing_tool=1
 fi

an paste output here

#3 Updated by Stefan Schöfegger over 1 year ago

  • Status changed from New to Triaged
  • Assignee set to Stefan Schöfegger
  • % Done changed from 0 to 20

#4 Updated by Stefan Schöfegger over 1 year ago

Please also send the output of

exiv2 -g Exif.Photo.ISOSpeedRatings -Pt P1110095.rw2
exiv2 --version

Which distro do you use?

#5 Updated by Alex Stepanov over 1 year ago

$ ls -al /bin/sh
lrwxrwxrwx 1 root root 4 Oct 30 02:19 /bin/sh -> bash*

@$ ~/darktable/tools/noise/gen-profile -d .
===> Check for required tools
--> Check shell: /usr/bin/fish version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
debug iso: RW2 IMAGE 160
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution
@

#6 Updated by Alex Stepanov over 1 year ago

Which distro do you use?

$ uname -a
Linux 4.14.14-1-MANJARO #1 SMP PREEMPT Wed Jan 17 22:34:54 UTC 2018 x86_64 GNU/Linux

https://manjaro.org/
(Arch- based Distro)

$ exiv2 -g Exif.Photo.ISOSpeedRatings -Pt P1110095.rw2
exiv2 --version

RW2 IMAGE
160
exiv2 0.26 001a00 (64 bit build)
Copyright (C) 2004-2017 Andreas Huggel.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public
License along with this program; if not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA

#7 Updated by Stefan Schöfegger over 1 year ago

I think it's an exiv2 0.26 issue. It should return 160 not

RW2 IMAGE
160

Can you test downgrading to exiv2 0.25? In the changelog of exiv2 I found nothing that suggest such a change. http://www.exiv2.org/changelog.html

#8 Updated by Alex Stepanov over 1 year ago

I changed the shell from the fish to the bash... but nothing changed!

~/darktable/tools/noise/gen-profile -d .
===> Check for required tools
--> Check shell: /bin/bash version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
debug iso: RW2 IMAGE 160
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution

I think it's an exiv2 0.26 issue. It should return 160 not
Ok, i will try.

#9 Updated by Alex Stepanov over 1 year ago

$exiv2 --version

exiv2 0.25 001900 (64 bit build)
Copyright (C) 2004-2015 Andreas Huggel.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public
License along with this program; if not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA

$ ~/darktable/tools/noise/gen-profile -d .

===> Check for required tools
--> Check shell: /bin/bash version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 160 image: ./P1110094.rw2
debug iso: 160
--> Found ISO 160 image: ./P1110095.rw2
debug iso: 160
--> Found ISO 200 image: ./P1110096.rw2
debug iso: 200
--> Found ISO 250 image: ./P1110097.rw2
debug iso: 250
--> Found ISO 320 image: ./P1110098.rw2
debug iso: 320
--> Found ISO 400 image: ./P1110099.rw2
debug iso: 400
--> Found ISO 500 image: ./P1110100.rw2
debug iso: 500
--> Found ISO 640 image: ./P1110101.rw2
debug iso: 640
--> Found ISO 800 image: ./P1110102.rw2
debug iso: 800
--> Found ISO 1000 image: ./P1110103.rw2
debug iso: 1000
--> Found ISO 1250 image: ./P1110104.rw2
debug iso: 1250
--> Found ISO 1600 image: ./P1110105.rw2
debug iso: 1600
--> Found ISO 2000 image: ./P1110106.rw2
debug iso: 2000
--> Found ISO 2500 image: ./P1110107.rw2
debug iso: 2500
--> Found ISO 3200 image: ./P1110108.rw2
debug iso: 3200
--> Found ISO 4000 image: ./P1110109.rw2
debug iso: 4000
--> Found ISO 5000 image: ./P1110110.rw2
debug iso: 5000
--> Found ISO 6400 image: ./P1110111.rw2
debug iso: 6400
--> Found ISO 8000 image: ./P1110112.rw2
debug iso: 8000
--> Found ISO 10000 image: ./P1110113.rw2
debug iso: 10000
--> Found ISO 12800 image: ./P1110114.rw2
debug iso: 12800

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 160:
    ./P1110094.rw2

Yeah! Thank you so much! Indeed, the problem was in packet exiv2 version 2.6!
You were right! Requires rollback to version 2.5 for this package

#10 Updated by Alex Stepanov over 1 year ago

It turned out not so simple ...
There is no previous error with exiv2 version 2.5 and the script starts working. But stops working Darktable!

$ darktable
darktable: error while loading shared libraries: libexiv2.so.26: cannot open shared object file: No such file or directory

In the end, I still can not get the profile for the 'denoise (profiled)' module in dt

#11 Updated by Alex Stepanov over 1 year ago

I sent a bug out to the developer exiv2. Has already received the answer. The developer submit this fix later today.

I got the correct profile for 'denoise (profiled)' module by this way:
>> subr.sh <<

get_exif_key() {
    local file key
    file=$1
    key=$2

    exiv2 -g "$key" -Pt "$file" 2>/dev/null | sed -e '1d' || :
}

Yes, is a very dirty hack, but working!

I created several sets of shots for profiling. I send the best (optimal) profile.

#12 Updated by Stefan Schöfegger over 1 year ago

I sent a bug out to the developer exiv2

Thank you. I set up a chroot test environment to test exiv2 0.26 to reproduce this bug. I'll have a look on your profile later.

#13 Updated by Stefan Schöfegger over 1 year ago

For the sake of completeness:
http://dev.exiv2.org/issues/1334

#14 Updated by Stefan Schöfegger over 1 year ago

  • % Done changed from 20 to 0
  • Status changed from Triaged to Closed: upstream

Profile is added, thank you.
I close this issue with closed:upstream as the main issue must be fixed in exiv2.

#15 Updated by Roman Lebedev over 1 year ago

  • Target version set to 2.6.0

Also available in: Atom PDF