Project

General

Profile

Camera Support #11999

Uncompressed 16-bit DNG's are broken.

Added by a dg over 1 year ago. Updated over 1 year ago.

Status:
Fixed
Priority:
Medium
Assignee:
Category:
General
Target version:
Start date:
02/06/2018
Due date:
% Done:

100%

Affected Version:
git master branch

Description

hi,
i would like to report the following issue:
- i have been using darktable since 2014 and since 2016 with a leica m240 and i could always import my raws from this camera (and an olympus OMD and later PenD), always using the dev. version with ppa for apt-get.
- since upgrading to 2.4.1 all my dngs from the leica have the same issue (see attached screenshot), even the ones from before with amounts of post-processing done to them.

i added a screenshot and the xmp file, in case.
i can upload anything more if necessary. however, a leica dng is 40ishM so with my flaky internet connection it needs to be planned. i am not a computer wizard but i have reasonable computer skills so just let me know if there is anything you need me to do.

note: i apologize for quitting without warning the irc channel, my connection is, as said, sketchy.
all the best,
a-

Screenshot_2018-02-06_12-03-11.png (1.12 MB) a dg, 02/06/2018 01:57 PM

L1006368.DNG.xmp (2.29 KB) a dg, 02/06/2018 02:01 PM

Associated revisions

Revision 343782b8
Added by Roman Lebedev over 1 year ago

RawSpeed submodule update: some stuff

Fixes #11998
Fixes #11999

History

#1 Updated by Roman Lebedev over 1 year ago

  • Tracker changed from Bug to Camera Support
  • Status changed from New to Incomplete
  • % Done changed from 0 to 20

Works fine here with 2.4.1 + https://raw.pixls.us/getfile.php/649/raw/RAW_LEICA_M240.DNG
I'm not sure what could be the problem.
Upload the sample please (and, please do contribute some sample from that camera to raw.pixls.us, since we do not have CC0 sample for this camera.)

I'd try stashing ~/.config/darktable/ away, and importing an image into clean db (or use new user).

#2 Updated by a dg over 1 year ago

hi roman,
i tried to purge the db but to no avail.
and indeed the one file you provided works as well on darktable 2.4.1 (but with a thumbnail in black and white for some reason...)

to be noted that i can work on the dngs (e.g. the one from the screenshot) with gimp or rawtherapy so the file in itself should not be corrupted or anything.

i will have to find a raw file smaller that the one i used in the original post's screenshot because i cannot upload it here at the moment.

(and i will try to contribute to pixls.us yes. looking into it.
thanks for your answer... it is really weird what's happening.
best,
a-

#3 Updated by Roman Lebedev over 1 year ago

a dg wrote:

hi roman,
i tried to purge the db but to no avail.
and indeed the one file you provided works as well on darktable 2.4.1 (but with a thumbnail in black and white for some reason...)

to be noted that i can work on the dngs (e.g. the one from the screenshot) with gimp or rawtherapy so the file in itself should not be corrupted or anything.

Well, sounds weird.
Are those dng's straight out-of-camera, or were they modified in any way (digikam/exiftool/etc) ?

i will have to find a raw file smaller that the one i used in the original post's screenshot because i cannot upload it here at the moment.

You could upload it elsewhere and post the link.

(and i will try to contribute to pixls.us yes. looking into it.
thanks for your answer... it is really weird what's happening.
best,
a-

#4 Updated by a dg over 1 year ago

to be noted that i can work on the dngs (e.g. the one from the screenshot) with gimp or rawtherapy so the file in itself should not be corrupted or anything.

Well, sounds weird.
Are those dng's straight out-of-camera, or were they modified in any way (digikam/exiftool/etc) ?

they are straight out-of-camera, i usually dump the content of my card on the computer and then import directories into darktable.
i shipped the dng towards you, i hope it is fine. it's all i can do right now.

a-

#5 Updated by Roman Lebedev over 1 year ago

@a dg please do contribute a sample to RPU.
It is strange that it regressed.

737c7238c8921f9c5536aaf4538591dbaac125a9 is the first bad commit
commit 737c7238c8921f9c5536aaf4538591dbaac125a9
Author: Roman Lebedev <lebedev.ri@gmail.com>
Date:   Wed Sep 13 18:42:22 2017 +0300

    AbstractDngDecompressor and friends: operate on ByteStream

    Passing offset+size must die.

:040000 040000 668f0fa6d4586d8747c7a0bd849da3de9cf35378 e8802614edd8a1abea707896379188a192ca7e01 M      src

darktable/src/external/rawspeed$ git bisect log
git bisect start
# bad: [91e6836f80ed55c7c093d503c083ec66fa219904] Actually properly add Panasonic DMC-ZS100 support.
git bisect bad 91e6836f80ed55c7c093d503c083ec66fa219904
# good: [11f079c873f6202fd46bdd0e2dd6d63489d06c5e] Add CMake build system.
git bisect good 11f079c873f6202fd46bdd0e2dd6d63489d06c5e
# skip: [e5530fcff4ff75f9e354b9597dd17703f864b6e2] Make most "protected" member variables "private" 
git bisect skip e5530fcff4ff75f9e354b9597dd17703f864b6e2
# good: [8ca0b46d538e12df132beee9750f0c1fe090d761] CMake: add basic plumbing to simplify running rstest on rpu samples
git bisect good 8ca0b46d538e12df132beee9750f0c1fe090d761
# bad: [fa7127f9cf75a536efa77ac4f911a4c808f0cb0b] Fuzz: libFuzzer_dummy_main: ignore -help=1, to be compatible with libFuzzer
git bisect bad fa7127f9cf75a536efa77ac4f911a4c808f0cb0b
# good: [8ee3d761393065b6d35f2ff9fef3a5c1c3c08f7d] AbstractLJpegDecompressor::parseSOF(): slight cleanup
git bisect good 8ee3d761393065b6d35f2ff9fef3a5c1c3c08f7d
# bad: [9d4f2d45e40d8fba618a7290e79b85305ca79315] CMake: refactor gcc code coverage report too
git bisect bad 9d4f2d45e40d8fba618a7290e79b85305ca79315
# bad: [04abde06721d0b6c19880c03f4bd059b8c2e42f9] DngDecoder::decodeData(): tiles: some cleanup
git bisect bad 04abde06721d0b6c19880c03f4bd059b8c2e42f9
# bad: [0bd1bb38c6998ad641e12921daccc6298a684591] RafDecoder::isCompressed(): don't forget to sanitize image dimensions
git bisect bad 0bd1bb38c6998ad641e12921daccc6298a684591
# bad: [bc859b6d4352327479be5658629cf3be176b2047] Cr2Decompressor/Cr2Decoder: operate on ByteStream
git bisect bad bc859b6d4352327479be5658629cf3be176b2047
# good: [18c5f109df78ca4115e5db3023c0c2d37f67a879] AbstractDngDecompressor: kill unused nThreads variable
git bisect good 18c5f109df78ca4115e5db3023c0c2d37f67a879
# bad: [1c5b6691f545d3c71c930c4e9993a6c3482a7d57] LJpegDecompressor: kill non-ByteStream constructors
git bisect bad 1c5b6691f545d3c71c930c4e9993a6c3482a7d57
# bad: [737c7238c8921f9c5536aaf4538591dbaac125a9] AbstractDngDecompressor and friends: operate on ByteStream
git bisect bad 737c7238c8921f9c5536aaf4538591dbaac125a9
# first bad commit: [737c7238c8921f9c5536aaf4538591dbaac125a9] AbstractDngDecompressor and friends: operate on ByteStream

#6 Updated by Roman Lebedev over 1 year ago

Ah right, there are two raw types on that camera - compressed and uncompressed.
The RPU sample we have is compressed, and it is fine, and yours is uncompressed, and it broke as i have guessed in https://github.com/darktable-org/rawspeed/commit/737c7238c8921f9c5536aaf4538591dbaac125a9#diff-734eaf4742505075f75eadd3968728e8R78

So, @a dg, please do contribute two raw samples (the camera and scene should be static, so the content should be the same):
  • compressed raw
  • uncompressed raw

After that i will fix it.

Thank you for reporting this!

#7 Updated by Roman Lebedev over 1 year ago

  • Priority changed from Low to Medium
  • Assignee set to Roman Lebedev

#8 Updated by Roman Lebedev over 1 year ago

  • Subject changed from import of .dng from leica m240 not working since 2.4.1 to Uncompressed 16-bit DNG's are broken.

#9 Updated by a dg over 1 year ago

So, @a dg, please do contribute two raw samples (the camera and scene should be static, so the content should be the same):
  • compressed raw
  • uncompressed raw

just did it. let me know if all went fine.
a-

#10 Updated by Roman Lebedev over 1 year ago

  • Status changed from Incomplete to Triaged
  • Affected Version changed from 2.4.1 to git master branch

a dg wrote:

So, @a dg, please do contribute two raw samples (the camera and scene should be static, so the content should be the same):
  • compressed raw
  • uncompressed raw

just did it. let me know if all went fine.
a-

Yep, thank you, samples received.

#11 Updated by Roman Lebedev over 1 year ago

  • % Done changed from 20 to 50
  • Status changed from Triaged to In Progress

#12 Updated by Roman Lebedev over 1 year ago

  • % Done changed from 50 to 100
  • Status changed from In Progress to Fixed

#13 Updated by Roman Lebedev over 1 year ago

  • Target version set to 2.6.0

Also available in: Atom PDF