Project

General

Profile

Bug #11963

Problem with profiling sensor and photon noise for Panasonic DMC-GF6 camera

Added by Alex Stepanov 12 months ago. Updated 12 months ago.

Status:
Closed: upstream
Priority:
Low
Category:
-
Target version:
Start date:
01/23/2018
Due date:
% Done:

0%

Affected Version:
git master branch
System:
other GNU/Linux
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Hello!
I need some help...

I found information how to profile my camera in this place:
https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/
My Panasonic DMC-GF6 camera hasn't a profile for 'denoise (profiled)' module in Darktable.

I could not find the directories and executable files specified in the manual, so I had to download all from branch master https://github.com/darktable-org/darktable.git

First of all, I tried the script on RAW files from the Canon EOD 650d camera and the script worked perfectly! I got a profile, graphs in PDF format and a full set of files. All is OK!

The next step was to do the same for Panasonic DMC-GF6. In accordance with the instruction, I took one shot for each ISO speed with a defocus. After this i run
~/darktable/tools/noise/gen-profile -d .
But, this time, the script does not work until the end and stops with an error.
The problem begins from this line in subr.sh file:
iso=$(get_image_iso "$image")

Full output to console:
@~/darktable/tools/noise/gen-profile -d .

===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution@

I delete all files from the Panasonic camera and replace they (in the same place) by files from the Canon. I started the same script again and it worked successfully.
That is, the problem somehow concerns the RAW files ONLY from the Panasonic DMC-GF6 camera

P1110095.rw2 (18.5 MB) Alex Stepanov, 01/27/2018 04:38 PM

dt-noiseprofile-20180128.tar.gz - Optimal denoise profile for Panasonic DMC-GF6 camera (23 MB) Alex Stepanov, 01/28/2018 04:49 PM

Associated revisions

Revision a5703721
Added by Stefan Schöfegger 12 months ago

noise profile: Add Panasonic DMC-GF6, refs #11963

Revision 7f573429
Added by Stefan Schöfegger 11 months ago

noise profile: Add Panasonic DMC-GF6, refs #11963

(cherry picked from commit a5703721842c126e95ac3a3c58ccdbf0730a5cc6)

History

#1 Updated by Alex Stepanov 12 months ago

One sample image from a Panasonic DMC-GF6 camera that causes an error

#2 Updated by Stefan Schöfegger 12 months ago

Your image does not trigger an error on my pc. This error is reported a few times now. It is fixed for Nikon in #11580, I have a look into it.

What does

ls -al /bin/sh

output?

Please add one debug message

--- a/tools/noise/gen-profile
+++ b/tools/noise/gen-profile
@@ -65,6 +65,8 @@ fi
 echo "===> Check for required tools" 
 missing_tool=0

+echo "--> Check shell: $SHELL" 
+
 if ! image_info_tools_installed; then
        missing_tool=1
 fi

an paste output here

#3 Updated by Stefan Schöfegger 12 months ago

  • Status changed from New to Triaged
  • Assignee set to Stefan Schöfegger
  • % Done changed from 0 to 20

#4 Updated by Stefan Schöfegger 12 months ago

Please also send the output of

exiv2 -g Exif.Photo.ISOSpeedRatings -Pt P1110095.rw2
exiv2 --version

Which distro do you use?

#5 Updated by Alex Stepanov 12 months ago

$ ls -al /bin/sh
lrwxrwxrwx 1 root root 4 Oct 30 02:19 /bin/sh -> bash*

@$ ~/darktable/tools/noise/gen-profile -d .
===> Check for required tools
--> Check shell: /usr/bin/fish version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
debug iso: RW2 IMAGE 160
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution
@

#6 Updated by Alex Stepanov 12 months ago

Which distro do you use?

$ uname -a
Linux 4.14.14-1-MANJARO #1 SMP PREEMPT Wed Jan 17 22:34:54 UTC 2018 x86_64 GNU/Linux

https://manjaro.org/
(Arch- based Distro)

$ exiv2 -g Exif.Photo.ISOSpeedRatings -Pt P1110095.rw2
exiv2 --version

RW2 IMAGE
160
exiv2 0.26 001a00 (64 bit build)
Copyright (C) 2004-2017 Andreas Huggel.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public
License along with this program; if not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA

#7 Updated by Stefan Schöfegger 12 months ago

I think it's an exiv2 0.26 issue. It should return 160 not

RW2 IMAGE
160

Can you test downgrading to exiv2 0.25? In the changelog of exiv2 I found nothing that suggest such a change. http://www.exiv2.org/changelog.html

#8 Updated by Alex Stepanov 12 months ago

I changed the shell from the fish to the bash... but nothing changed!

~/darktable/tools/noise/gen-profile -d .
===> Check for required tools
--> Check shell: /bin/bash version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO RW2 IMAGE 160 image: ./P1110094.rw2
debug iso: RW2 IMAGE 160
/home/alex/darktable/tools/noise/subr.sh: line 22: ${var_images_RW2 IMAGE 160}: bad substitution

I think it's an exiv2 0.26 issue. It should return 160 not
Ok, i will try.

#9 Updated by Alex Stepanov 12 months ago

$exiv2 --version

exiv2 0.25 001900 (64 bit build)
Copyright (C) 2004-2015 Andreas Huggel.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public
License along with this program; if not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA

$ ~/darktable/tools/noise/gen-profile -d .

===> Check for required tools
--> Check shell: /bin/bash version: 4.4.12(1)-release.
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for internal tools availability
--> Build profiling tools
make: Entering directory '/home/alex/darktable/tools/noise'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/alex/darktable/tools/noise'
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 160 image: ./P1110094.rw2
debug iso: 160
--> Found ISO 160 image: ./P1110095.rw2
debug iso: 160
--> Found ISO 200 image: ./P1110096.rw2
debug iso: 200
--> Found ISO 250 image: ./P1110097.rw2
debug iso: 250
--> Found ISO 320 image: ./P1110098.rw2
debug iso: 320
--> Found ISO 400 image: ./P1110099.rw2
debug iso: 400
--> Found ISO 500 image: ./P1110100.rw2
debug iso: 500
--> Found ISO 640 image: ./P1110101.rw2
debug iso: 640
--> Found ISO 800 image: ./P1110102.rw2
debug iso: 800
--> Found ISO 1000 image: ./P1110103.rw2
debug iso: 1000
--> Found ISO 1250 image: ./P1110104.rw2
debug iso: 1250
--> Found ISO 1600 image: ./P1110105.rw2
debug iso: 1600
--> Found ISO 2000 image: ./P1110106.rw2
debug iso: 2000
--> Found ISO 2500 image: ./P1110107.rw2
debug iso: 2500
--> Found ISO 3200 image: ./P1110108.rw2
debug iso: 3200
--> Found ISO 4000 image: ./P1110109.rw2
debug iso: 4000
--> Found ISO 5000 image: ./P1110110.rw2
debug iso: 5000
--> Found ISO 6400 image: ./P1110111.rw2
debug iso: 6400
--> Found ISO 8000 image: ./P1110112.rw2
debug iso: 8000
--> Found ISO 10000 image: ./P1110113.rw2
debug iso: 10000
--> Found ISO 12800 image: ./P1110114.rw2
debug iso: 12800

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 160:
    ./P1110094.rw2

Yeah! Thank you so much! Indeed, the problem was in packet exiv2 version 2.6!
You were right! Requires rollback to version 2.5 for this package

#10 Updated by Alex Stepanov 12 months ago

It turned out not so simple ...
There is no previous error with exiv2 version 2.5 and the script starts working. But stops working Darktable!

$ darktable
darktable: error while loading shared libraries: libexiv2.so.26: cannot open shared object file: No such file or directory

In the end, I still can not get the profile for the 'denoise (profiled)' module in dt

#11 Updated by Alex Stepanov 12 months ago

I sent a bug out to the developer exiv2. Has already received the answer. The developer submit this fix later today.

I got the correct profile for 'denoise (profiled)' module by this way:
>> subr.sh <<

get_exif_key() {
    local file key
    file=$1
    key=$2

    exiv2 -g "$key" -Pt "$file" 2>/dev/null | sed -e '1d' || :
}

Yes, is a very dirty hack, but working!

I created several sets of shots for profiling. I send the best (optimal) profile.

#12 Updated by Stefan Schöfegger 12 months ago

I sent a bug out to the developer exiv2

Thank you. I set up a chroot test environment to test exiv2 0.26 to reproduce this bug. I'll have a look on your profile later.

#13 Updated by Stefan Schöfegger 12 months ago

For the sake of completeness:
http://dev.exiv2.org/issues/1334

#14 Updated by Stefan Schöfegger 12 months ago

  • % Done changed from 20 to 0
  • Status changed from Triaged to Closed: upstream

Profile is added, thank you.
I close this issue with closed:upstream as the main issue must be fixed in exiv2.

#15 Updated by Roman Lebedev 12 months ago

  • Target version set to 2.6.0

Also available in: Atom PDF