Project

General

Profile

Bug #11398

Compressed DNGs from Leica M9 doesn't work

Added by Pontus Adefjord over 2 years ago. Updated over 2 years ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
General
Target version:
Start date:
12/28/2016
Due date:
% Done:

100%

Affected Version:
2.2.0
System:
other GNU/Linux
bitness:
64-bit
hardware architecture:
amd64/x86

Description

When I try to open a compressed DNG in darkroom, I get the error message "Failed to read camera white balance". Trying to set the white balance manually doesn't help, and when I go back to lighttable it says "Image [filename].DNG not available". Uncompressed DNGs work, and compressed have worked fine until I updated. Running 2.2 on Arch Linux.

L1004548.DNG (17.5 MB) Pontus Adefjord, 12/28/2016 11:32 AM

L1004870.DNG (34.7 MB) Pontus Adefjord, 01/02/2017 03:47 PM

Associated revisions

Revision 8b0a8e7d
Added by Roman Lebedev over 2 years ago

Rawspeed submodule update: large rework, better code. Fixes #11398

History

#1 Updated by Roman Lebedev over 2 years ago

Compressed DNGs

Are you sure that dng is compressed?

$ exiv2 -Pkt L1004548.DNG | grep -i compression
L1004548.DNG: (No XMP data found in the file)
Exif.Image.Compression                        Uncompressed
Exif.SubImage1.Compression                    Uncompressed

That being said, it fails to load indeed.

#2 Updated by Pontus Adefjord over 2 years ago

It is compressed, weird that the exif says otherwise. Maybe that's the problem, perhaps I could try to change these values and see if the files work. Still, they used to work fine in earlier versions.

#3 Updated by Pontus Adefjord over 2 years ago

From what I can find online, DNG supports compression 1 (uncompressed) or 7 (jpeg). I tried setting Exif.Image.Compression to 7, but darktable still can't read the file. For some reason I can't seem to change Exif.SubImage1.Compression using exiv2.

#4 Updated by Roman Lebedev over 2 years ago

Pontus Adefjord wrote:

From what I can find online, DNG supports compression 1 (uncompressed) or 7 (jpeg).

I tried setting Exif.Image.Compression to 7, but darktable still can't read the file. For some reason I can't seem to change Exif.SubImage1.Compression using exiv2.

Don't EVER modify raw files. EVER.

Is that attached L1004548.DNG is directly from camera, or was it modified by some software, like exiftool/exiv2/etc?

#5 Updated by Pontus Adefjord over 2 years ago

Ok :) The attached file is directly from camera.

#6 Updated by Pontus Adefjord over 2 years ago

Here's an uncompressed DNG. No problems at all with this one.

#7 Updated by Licaon Kter over 2 years ago

Did this ever work? I mean lossless compressed DNGs.

I raised this issue back in 1.6.3, albeit in comments:
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43280
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43321

Testing now with the samples Magic Lantern user Danne posted for their work in progress in camera lossless compressed DNG saving (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176809#msg176809):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG

65260001.DNG          Exif.Image.Compression                        Uncompressed
65260001.DNG          Exif.SubImage1.Compression                    JPEG
65260002.DNG          Exif.Image.Compression                        Uncompressed
65260002.DNG          Exif.SubImage1.Compression                    JPEG

I get the same result like in my 1.6.3 posts above.

#8 Updated by Roman Lebedev over 2 years ago

Licaon Kter wrote:

Did this ever work? I mean lossless compressed DNGs.

I did not check. I have some wip code for rawspeed to support HDR fp32 DNG's, this may be one more DNG feature that is not yet supported.

I raised this issue back in 1.6.3, albeit in comments:
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43280
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43321

Testing now with the samples Magic Lantern user Danne posted for their work in progress in camera lossless compressed DNG saving (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176809#msg176809):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG
[...]

BTW, are these proper dng, with actual color matrix and stuff?
If not, then they are no better than early CHDK, and are being mis-advertised...

I get the same result like in my 1.6.3 posts above.

#9 Updated by Licaon Kter over 2 years ago

You can check them out, I guess you mean these:

65260001.DNG          Exif.Image.ColorMatrix1                       6722/10000 -635/10000 -963/10000 -4287/10000 12460/10000 2028/10000 -908/10000 2162/10000 5668/10000
65260002.DNG          Exif.Image.ColorMatrix1                       6722/10000 -635/10000 -963/10000 -4287/10000 12460/10000 2028/10000 -908/10000 2162/10000 5668/10000

Again, LightZone and RawTherapee open them ok, does Darktable have some other requirements?

If not, I guess "lossless compressed DNGs" are just not (yet) supported then?

#10 Updated by Roman Lebedev over 2 years ago

Licaon Kter wrote:

You can check them out, I guess you mean these:
[...]

I was only talking about ML DNG's there.

Again, LightZone and RawTherapee open them ok, does Darktable have some other requirements?

If not, I guess "lossless compressed DNGs" are just not (yet) supported then?

That was my guess in https://redmine.darktable.org/issues/11398#note-8

#11 Updated by Pontus Adefjord over 2 years ago

I don't know anything about Magic Lantern, but compressed DNGs from my M9 worked fine until 2.2.

#12 Updated by Eildert Groeneveld over 2 years ago

Roman Lebedev wrote:

Licaon Kter wrote:

Did this ever work? I mean lossless compressed DNGs.

sure did. I have been using DT for ages on my M9 DNGs

I did not check. I have some wip code for rawspeed to support HDR fp32 DNG's, this may be one more DNG feature that is not yet supported.

I raised this issue back in 1.6.3, albeit in comments:
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43280
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43321

Testing now with the samples Magic Lantern user Danne posted for their work in progress in camera lossless compressed DNG saving (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176809#msg176809):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG
[...]

BTW, are these proper dng, with actual color matrix and stuff?
If not, then they are no better than early CHDK, and are being mis-advertised...

I get the same result like in my 1.6.3 posts above.

#13 Updated by Roman Lebedev over 2 years ago

Eildert Groeneveld wrote:

Roman Lebedev wrote:

Licaon Kter wrote:

Did this ever work? I mean lossless compressed DNGs.

sure did. I have been using DT for ages on my M9 DNGs

Some hardening of rawspeed broke it.
Probably because there were no such sample in the test set.
This is why it is important for EVERYONE to visit https://raw.pixls.us/ and make sure that we have all the needed samples from the camera he owns, under CC0.
This way, hopefully, it won't happen again in future.

I did not check. I have some wip code for rawspeed to support HDR fp32 DNG's, this may be one more DNG feature that is not yet supported.

I raised this issue back in 1.6.3, albeit in comments:
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43280
http://www.darktable.org/2015/03/released-darktable-1-6-3/#comment-43321

Testing now with the samples Magic Lantern user Danne posted for their work in progress in camera lossless compressed DNG saving (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176809#msg176809):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260001.DNG
https://bitbucket.org/Dannephoto/magic-lantern/downloads/65260002.DNG
[...]

BTW, are these proper dng, with actual color matrix and stuff?
If not, then they are no better than early CHDK, and are being mis-advertised...

I get the same result like in my 1.6.3 posts above.

#14 Updated by Eildert Groeneveld over 2 years ago

Roman Lebedev wrote:

Eildert Groeneveld wrote:

Roman Lebedev wrote:

Licaon Kter wrote:

Did this ever work? I mean lossless compressed DNGs.

sure did. I have been using DT for ages on my M9 DNGs

Some hardening of rawspeed broke it.
Probably because there were no such sample in the test set.
This is why it is important for EVERYONE to visit https://raw.pixls.us/ and make sure that we have all the needed samples from the camera he owns, under CC0.
This way, hopefully, it won't happen again in future.

Roman, this ounds like you found the bug? I cloned dt today and did
git submodule init
git submodule update

compilation when through OK, but I still get:

[temperature] failed to read camera white balance information from `L1048787.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'
Please consider providing samples on <https://raw.pixls.us/&gt;, thanks!
[rawspeed] (L1048810.DNG) DNG decoder: IO error occurred in first slice, unable to decode more. Error is: Out of buffer read
[temperature] failed to read camera white balance information from `L1048810.DNG'!
[temperature] failed to read camera white balance information from `L1048810.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'
Please consider providing samples on <https://raw.pixls.us/&gt;, thanks!
[rawspeed] (L1048811.DNG) DNG decoder: IO error occurred in first slice, unable to decode more. Error is: Out of buffer read

is there anything that I can do other than uploading a compressed and uncompressed DNG to pixls (which I have done)?

#15 Updated by Tobias Ellinghaus over 2 years ago

Try "git submodule foreach git pull origin develop" and build again.

#16 Updated by Roman Lebedev over 2 years ago

Eildert Groeneveld wrote:

Roman, this ounds like you found the bug? I cloned dt today and did
git submodule init
git submodule update

compilation when through OK, but I still get:

[temperature] failed to read camera white balance information from `L1048787.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'
Please consider providing samples on <https://raw.pixls.us/&gt;, thanks!
[rawspeed] (L1048810.DNG) DNG decoder: IO error occurred in first slice, unable to decode more. Error is: Out of buffer read
[temperature] failed to read camera white balance information from `L1048810.DNG'!
[temperature] failed to read camera white balance information from `L1048810.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'
Please consider providing samples on <https://raw.pixls.us/&gt;, thanks!
[rawspeed] (L1048811.DNG) DNG decoder: IO error occurred in first slice, unable to decode more. Error is: Out of buffer read

That is fine.
The support is not fixed.
And it talks about the rawspeed's internal database, not about raw.pixls.us.
So all is within expected results.

is there anything that I can do other than uploading a compressed and uncompressed DNG to pixls (which I have done)?

Thank you for uploading the samples, but i'm not sure you can do anything els.

#17 Updated by Eildert Groeneveld over 2 years ago

Tobias Ellinghaus wrote:

Try "git submodule foreach git pull origin develop" and build again.

Roman, done it. Same result :(

#18 Updated by Roman Lebedev over 2 years ago

Hmm, should be fixed now, surprisingly.

#19 Updated by Roman Lebedev over 2 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

#20 Updated by Eildert Groeneveld over 2 years ago

Roman Lebedev wrote:

Hmm, should be fixed now, surprisingly.

Roman, you are moving fast!!I did a git pull and a new build and install.

Only, the problem is still around:

[temperature] failed to read camera white balance information from `L1048810.DNG'!
[temperature] failed to read camera white balance information from `L1048810.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'

Has the M9 disappeared altogether?

#21 Updated by Roman Lebedev over 2 years ago

Eildert Groeneveld wrote:

Only, the problem is still around:

[temperature] failed to read camera white balance information from `L1048810.DNG'!
[temperature] failed to read camera white balance information from `L1048810.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'

Just ignore it. It simply says that this camera is not listed in
https://github.com/darktable-org/rawspeed/blob/develop/data/cameras.xml

#22 Updated by Eildert Groeneveld over 2 years ago

Roman Lebedev wrote:

Eildert Groeneveld wrote:

Only, the problem is still around:

[temperature] failed to read camera white balance information from `L1048810.DNG'!
[temperature] failed to read camera white balance information from `L1048810.DNG'!
RawSpeed:Unable to find camera in database: 'Leica Camera AG' 'M9 Digital Camera' 'dng'

Just ignore it. It simply says that this camera is not listed in
https://github.com/darktable-org/rawspeed/blob/develop/data/cameras.xml

Alright, then everything is ok? (seems to be working)

In your url I see the M8 (APS C) but not the M9 (FF); does that matter?

<Camera make="Leica Camera AG" model="M8 Digital Camera" mode="dng">
<ID make="Leica" model="M8">M8 Digital Camera</ID>

#23 Updated by Eildert Groeneveld over 2 years ago

hhmmm, there still seems to be problem: the thumbnails in Lighttable are those infamous skulls:

[temperature] failed to read camera white balance information from `L1048908.DNG'!
allocation failed???
[temperature] failed to read camera white balance information from `L1048908.DNG'!

which means (among others I guess) that I cannot switch to editing

The DNG does exists

#24 Updated by Roman Lebedev over 2 years ago

after updating darktable, you did not update the rawspeed git submodule.
you need to do something like: $ git submodule foreach git fetch && git submodule update -f
And after that, rebuild darktable.

#25 Updated by Eildert Groeneveld over 2 years ago

Roman Lebedev wrote:

after updating darktable, you did not update the rawspeed git submodule.
you need to do something like: $ git submodule foreach git fetch && git submodule update -f
And after that, rebuild darktable.

Thanks Roman, I have taken note for future reference.
From what I see: the skulls are gone. However, there seems to be another issue. Not sure it
is related to this bug. Under Image Information now the make Mamiya shows up instead of the
Leica stuff; on the console I get the following printout:

[imageop_init_presets] WARNING: Could not find versioning information for 'basecurve' preset 'M9 Digital Camera'
Until some is found, the preset will be unavailable.

also the same for tonecurve.

Further (probably as a result) presets under basecurve shows a deactivated M9 digital camera and an operational "Leica like"
The 'M9 Digital Camera' is in the presets, which I had put in, as upon import of a new RAW only a flat
basecurve got used. It seems that some pieces of info got lost last year.
some 5 years ago Pascal de Bruijn set the M9 up with color matrix, basecurve and all.

Is there a way that I can check what information needs to be available for a complete support of a camera?

#26 Updated by Roman Lebedev over 2 years ago

  • Target version set to 2.4.0

Also available in: Atom PDF