Project

General

Profile

Bug #9897

Darktable not reading lens info from the Pentax K-3 DNG files although its stored in the file.

Added by Jeff G over 5 years ago. Updated about 4 years ago.

Status:
Closed: invalid
Priority:
Low
Assignee:
-
Category:
General
Target version:
-
Start date:
04/11/2014
Due date:
% Done:

0%

Estimated time:
Affected Version:
git stable branch
System:
other GNU/Linux
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Just purchased a Pentax K-3 a few weeks ago and using my preferred raw format DNG. By using DNG you don't need to wait for your program of choice to be updated for a newer camera specific format and that's also good for compatibility across different programs or OS's.

Both Darktable 1.4.1 release and the git compiled version 32 or 64 bit appear to be reading all the information from DNG files EXCEPT the lens info. My K-3 is running the most current v1.03 firmware. The Pentax K-3 PEF raw files aren't yet supported apparently because when attempting to import/load them into DT I get the B/W skull. If you highlight the file the image info pane on the lower left of DT will show you the usual info embedded in the file, but again like the DNG’s no lens info.

Accompanied terminal ouput when attempting to load PEF’s:

[rawspeed] No decoder found. Sorry.
[imageio] '/home/retro/Pictures/K-3PEF/IMGP0878.PEF' blacklisted extension passed to libraw
/home/retro/Pictures/K-3PEF/IMGP0878.PEF: data corrupted at 32661814
[imageio] /home/retro/Pictures/K-3PEF/IMGP0878.PEF: Input/output error

PEF support would be nice, but I don't care so much about that since I prefer to use the wider excepted DNG raw format, but I need the lens info for auto corrections, sorting, tagging, rating, searching, etc...

The lens info is in there and read by exiftool under my distro Manjaro linux which is arch based. For what its worth its also read fine in Lightroom 5.3 under a virtualbox windows 7 environment. Is this a DT issue or a issue with one of the underlying dependencies that DT uses to read in the RAW formats? Whatever the case I hope its fixable. I uploaded a sample DNG from the camera and if needed I give provide a PEF from the same camera.

IMGP0158.DNG (32.1 MB) IMGP0158.DNG Jeff G, 04/11/2014 12:50 PM

History

#1 Updated by Christian Kanzian over 5 years ago

Such issues are very often reported, but not directly related to darktable.
I'm an ordinary user. However, darktable depends on exiv2 to read the metadata. Unluckily the exiv2 database is behind the well known exiftool.

If I do 'exiv2 -pt IMGP0158.DNG' no lens info is reported.

So it's not a darktable bug (close?) and should be reported to the exiv2 project.

By the way my Pentax DA 18-138/3.5-5.6 on a K-5 also isn't detected. In my case digikam, which I use for my JPGs also is effected of course.

#2 Updated by Jeff G over 5 years ago

  • Status changed from New to Closed: invalid

I will close it, but that's unfortunate darktable and many other photo programs rely on the less updated exiv2 project over the exiftool. Thanks for the confirmation and input. I'll see what info the exiv2 project has on these issues.

#3 Updated by Pascal de Bruijn over 5 years ago

  • Priority changed from Medium to Low

I meant to respond to this earlier, but here goes anyhow...

Looking at the IMGP0158.DNG file, it seems to be missing the Exif.Photo.LensModel, which is the nicest way to represent lens type data in EXIF.

Many vendors don't use that tag in their proprietary formats, because lenses often identify themselves as numbers, which need to be translated into lens names (though this could be done by the camera firmware of course, like some vendors do on newer bodies).

That lens number, is usually contained in the vendors proprietary EXIF blob called the MakerNote.

From what I can gather with a little bit of googling, there seem to be some issues with MakerNotes in DNG (though probably mostly related to how Adobe's own DNG converter handles them):
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/DNG.html

Aside from that it's not a shame we're not using ExifTool, because we can't in a sensible way. ExifTool is a Perl library/tool, which can't be sensibly linked into a C program. Exiv2 is realistically the only full featured open source library to work with EXIF data in a variety of (RAW) formats.

Now, I'm guessing Exiv2 isn't reading the Pentax proprietary MakerNote from the DNG, so I highly recommend filing a feature request with the Exiv2 folks.

#4 Updated by Jeff G over 5 years ago

Yeah I just made a new account on the exiv2 project and will make a bug report and inquiries with them. Its not only the DNG out of the Pentax K-3, but also it's proprietary PEF raw format which Exiv2 says they full support although no mention of the K-3, but obviously not the case with their latest camera the K-3.

#5 Updated by Pascal de Bruijn over 5 years ago

There is no such thing as "full" support for any RAW "format", because RAW "formats" aren't well defined in any sense. Besides being fully proprietary, they tend to vary from camera to camera... Sometimes with only minor differences, sometimes with major changes (endianness/compression/etc).

This is also why RawSpeed (what we use to read PEFs) doesn't support the K-3 yet, because Pentax made huge changes to it's format, one of which is changing the endianness IIRC.

#6 Updated by Jeff G over 5 years ago

That's why photo programs are often updated to handle file format changes and new equipment. I was hoping DNG being or trying to be a universal raw format would be fully supported and is with the exception of the lens info in regard to Pentax. As you said it might be an issue with the MakerNotes, but I've found a work around for that.

Adobe's Camera Raw 8.4 and DNG Converter 8.4 have no issue reading all of the info from the current PEF format. The converter is available for free, but relies on a minimum of Windows 7, no linux support. I briefly attempted to install DNG Converter via WINE, but it won't see the PEF files after you select them from within the program. I have a windows 7 virtualbox install I keep for things that really work in Windows only. Installing the converter in that and converting a few PEF format photos worked like a charm lens data and all! Its an extra step, but the DNG's it produces are fully read by Exiv2 and any program that depends on it. I've seen this in DT and Digikam at least.

I wanted to post this so Pentax end users like me a little frustrated over PEF and DNG issues know there is a work around of sorts.

#7 Updated by Pascal de Bruijn over 5 years ago

The DNG format is in most regards a near universal RAW format... That doesn't stop vendors from implementing it poorly... On the other hand of course, realistically having your camera body generate native DNGs at all is a bit of luxury compared to most other brands :)

Also keep in mind that RAW+EXIF are practically very complex formats, and converting them to DNG is by no means a trivial task, there is always risk of stuff breaking, so I'd suggest to keep the original PEFs around just in case.

#9 Updated by Emilian Bogatu over 4 years ago

exiv2 patched as per Bug#9124 (http://darktable.org/redmine/issues/9124 since maker in exif is Ricoh); as result exiv2 is returning the correct lens in Exif.PentaxDng.LensType (for both raw formats supported by K-3: DNG & PEF):

exiv2 -pt ./_IMG5285.PEF |grep Lens
Exif.PentaxDng.LensType Byte 4 smc PENTAX-DA 35mm F2.4 AL

However, darktable (git version/commit 9d77a28e54f8bc19592a170b1e4b9cf083f7b5a3 compiled with patched exiv2) is not using Exif.PentaxDng.LensType to extract lens information. As consequence:
- in left panel (Image information), lens section is not populated.
- in lens corrections - the lens need to be manually selected.

It seems that src/common/exif.cc should extract information from Exif.PentaxDng.LensType for Pentax DNG or PEF files.
Bellow patch to exif.cc seems to work properly but I'm not a programmer. Since the exiv2 field used in patch is Pentax (DNG/PEF) specific, I suppose that the current issue is not limited to K-3 but to all recent camera (even if maker is Pentax and not Ricoh in exif).
Please review the following patch:

diff --git a/src/common/exif.cc b/src/common/exif.cc
index 505a543..ca3f725 100644
--- a/src/common/exif.cc
++ b/src/common/exif.cc
@ -569,6 +569,10 @ static bool dt_exif_read_exif_data(dt_image_t *img, Exiv2::ExifData &exifData)
dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);
}
#endif
else if((pos = exifData.findKey(Exiv2::ExifKey("Exif.PentaxDng.LensType"))) != exifData.end() && pos->size())
+ {
+ dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);
+ }
else if((pos = Exiv2::lensName(exifData)) != exifData.end() && pos->size()) {
dt_strlcpy_to_utf8(img->exif_lens, sizeof(img->exif_lens), pos, exifData);

Also available in: Atom PDF

Go to top