Project

General

Profile

Feature #12501

Option to save metadata and tags in the RAW file (instead of a sidecar)

Added by Vincent Fregeac 6 months ago. Updated 6 months ago.

Status:
Closed: invalid
Priority:
Low
Assignee:
-
Category:
Lighttable
Target version:
-
Start date:
12/27/2018
Due date:
% Done:

0%

Affected Version:
git master branch
System:
all
bitness:
64-bit
hardware architecture:
amd64/x86

Description

With metadata and tags in sidecars, if the sidecar file is lost, all the information about the image is lost.

For someone who consider his images disposable, sidecars are fine: you make sure to backup the sidecar files with the RAW file so you know what your photos represent; the next generation will of course lose all the sidecar files but you don't care.

But, when you have discovered old photographic plates, you know the value of a sticker directly on the plate, to tell you that this kid was your grandfather when he was 7, and you also know that the notebook with the description of the photographic plates is never with the photographic plates. If it's not on the photographic plate, it's lost.

The modern equivalent is having the metadata in a sidecar file (it will be lost by the next generation) versus having them in the RAW file (if the image is not lost, the metadata is not lost).

Exiftool can write metadata in a lot of RAW files, including many proprietary format. We should have the option to write the metadata and tags directly in the RAW file (and we have a backup of course, so if it corrupts the raw file it's not that big of a problem). It should only apply to metadata and tags in my opinion, because we normally write them once, but not the development settings as they can be changed several time so writing them in the RAW file would multiply the chances of corrupting the RAW file.

History

#1 Updated by dar ix 6 months ago

AFAIK darktable has the premise

a) never to modify files
b) all edits go into sidecar files

There have been plenty of cases where tools would shred the raw files with the editing. Maybe as a counter proposal ... have the ability to override metadata from the file with metadata from the sidecar file. I have a few photos where the time on my camera was wrong and the image are sorted incorrectly now.

Also for your dnq conversion bug. given how much the devs discourage using the adobe tool for conversion, I dont see that happening either.

#2 Updated by Roman Lebedev 6 months ago

  • Status changed from New to Closed: invalid

No way in hell.

#3 Updated by Vincent Fregeac 6 months ago

Darktable premise is "non-destructive" which mean no information of the image is removed or altered. It doesn't mean no information can be appended. Also, I understand that when darktable was initially developed, Exiftool ability to add metadata to RAW files, especially proprietary formats, was still in beta and not really reliable. Time have passed since. I have saved metadata and tags in over 15,000 RAW files without a single glitch.

And there is a very good justification to save metadata and tags inside RAW files: As far as I know, very few, if any, Digital Asset Management solution accept or manage sidecar files, so with the metadata only in sidecar files, RAW files cannot be managed since darktable is not, yet, a good solution to manage a large collection of images.

Also, Roman, if you don't like that feature, no one will force you to use it if it's implemented. But, really, no one is even allowed to discuss it just because you don't like it. You mentioned in another issue how you don't like the dictatorship of Lightroom's workflow, you should be the first to understand the dictatorship of Roman Lebedev is no more justifiable. You should at least leave the feature request open to give the opportunity to other users and/or developers to discuss it. That's what open-source means, you don't own darktable, so it doesn't have to be only what you want or like.

#4 Updated by Roman Lebedev 6 months ago

Vincent Fregeac wrote:

Darktable premise is "non-destructive" which mean no information of the image is removed or altered.

You are mixing two completely different concepts.
"non-destructive" talks about the processing of the data, in the 32-bit floating-point pipeline, with every new step
producing a new "image", with an ability to losslessly go to any previous step and losslessly change things.
There is nothing about files/filesystem in that term.

It doesn't mean no information can be appended.

It does. darktable intentionally never ever opens source images for write.

Also, I understand that when darktable was initially developed, Exiftool ability to add metadata to RAW files, especially proprietary formats, was still in beta and not really reliable. Time have passed since.

exiv2 is used, not exiftool.

I have saved metadata and tags in over 15,000 RAW files without a single glitch.

OUCH. I really do hope you have backups.

There were numerous reports in recent years of digikam (==exiv2) breaking raw files by editing the metadata.
And by numerous i mean dozens. Nowadays digikam has learn the lesson, and no longer allows to write sidecars into raws.
https://bugs.kde.org/show_bug.cgi?id=387351 (yes, i can see that it has a "please shoot me into the foot" mode.)

And there is a very good justification to save metadata and tags inside RAW files: As far as I know, very few, if any, Digital Asset Management solution accept or manage sidecar files,

Sounds like a limitation in that software?

so with the metadata only in sidecar files, RAW files cannot be managed since darktable is not, yet, a good solution to manage a large collection of images.

Also, Roman, if you don't like that feature, no one will force you to use it if it's implemented. But, really, no one is even allowed to discuss it

Hm, we are disscussing it right now, i think?

just because you don't like it.

Not really just me though. This has been brought up a number of times, there are very specific reasons why it is a bad thing to do.

You mentioned in another issue how you don't like the dictatorship of Lightroom's workflow,

Hm, maybe?, i don't remember saying that, i don't remember not saying that.

you should be the first to understand the dictatorship of Roman Lebedev is no more justifiable.

You should at least leave the feature request open to give the opportunity to other users and/or developers to discuss it.

As it can be seen from the continuing comments, closing an issue does not automatically disallow posting new comments.

That's what open-source means, you don't own darktable, so it doesn't have to be only what you want or like.

#5 Updated by Vincent Fregeac 6 months ago

"darktable edits your images non-destructively all the way through its pipeline. Your original image is never modified!" dixit Darktable website itself. "Non-destructive" editing is not a concept invented by Darktable, and it always meant parametric editing, in opposition to direct pixel modification. Also, note that darktable website present "non-destructive" editing as not modifying the image, it doesn't talk about the file containing the image. 32bit-float computation was implemented for another reason: to maintain fine nuances of tone and color throughout the editing and avoid losing information until the final image is converted back to 16bits; that's a different concept than non-destructive editing. But the important point is, none of these concepts are about the file, they are about not modifying the image contained in the file.

When it comes to managing the RAW files, Digital Asset Management is not a software, it's an entire industry with dozens of entreprise-grade solutions plus a few hundreds of smaller, specialized solutions. And the requirement of DAM solutions to have metadata saved inside the file is not a limitation, it's one of the basic principles of digital asset management: informations outside the digital asset will be lost at one point; the only way to ensure that metadata will not be lost is to incorporate the metadata in the file itself. That's how metadata are saved in all digital assets, whether they are text documents, audio files, video files, as well as almost every image format except some recent, proprietary, RAW formats. The only reason the concept of sidecars was created is because, initially, there was no tool to save metadata inside proprietary RAW formats as it is done with every other format of every other type of document. Sidecars were and still are a temporary patch, not a long term solution. Even Darktable acknowledges that since, when exporting images in another format than the proprietary RAW format, it saves metadata as they should: inside the image file.

Now, you can decide that Darktable will do things differently than the rest of the industry, but you can't expect an entire DAM industry to adapt to darktable. The only thing you will achieve is to make darktable incompatible with everything else and living in its own silo. I don't think that was the intent of the developers who created darktable 8 years ago.

Now, if exiv2 is still not reliable today, maybe it would be time to consider using exiftool instead. According to exiv2 wiki itself: "Today Exiftool is amazingly complete, it reads and writes metadata, supports more image formats, more types of metadata and more Makernotes than Exiv2 and Andreas considers it the reference implementation for image metadata." So even the main developer of Exiv2 basically recommends to use exiftool rather than exiv2, except for applications where you need to read a huge volume of metadata in one batch. See for your self: http://dev.exiv2.org/projects/exiv2/wiki/How_does_Exiv2_compare_to_Exiftool. So, for the typical use of metadata in darktable, i.e. importing metadata from a few hundred images per batch, or modifying metadata of a few dozen image together, the "slow" exiftool will still be faster than the fastest human fingers selecting or typing the metadata. And, yes, exiftool, although written in Perl, does have a C/C++ interface.

As for having backup before modifying a RAW file, well, even using a "non-destructive" editor who doesn't even touch the RAW file without a backup would be insane. Beside, every tool I have used who save metadata inside RAW file create a backup file by default (generally using an extension like .xxx_original). Since I prefer to manage my backups outside of the image tools, I deactivate that option but, considering a lot of people don't even backup their documents, having this option activated by default is still a good practice.

As I said, time have passed since the early day of exif editing, i.e.:
- saving metadata in RAW files is not the gamble it was before;
- exiv2 is apparently not the preferred solution anymore (except huge import of metadata), even according to exiv2 developers.

So, maybe it's time to reconsider the initial assumption that modifying a RAW file has no benefit and is too risky anyway. The environment images live in has evolved in the past 10 years, as well as the tool used to handle the metadata in RAW files. I am not saying darktable should only save metadata inside RAW files, I understand that some people are still paranoid about modifying RAW files: saving metadata and tags inside RAW files should only be an option, not even activated by default, even with a big disclaimer if you want. But darktable should not live in its own silo, disregarding any notion of compatibility with the rest of the tools used in an image workflow. At least because that was not the intent of the original developers: darktable was supposed to be a FOSS alternative, aimed at enthousiasts and professionals, taking into account and implementing the current best/standard practice in an image workflow. So, except if darktable now targets beginner photographers, it should take into consideration the standard practice in modern digital image management. And one of the changes in the best practice of digital image management is to consider RAW images as digital assets, not only the developed images. But it requires RAW files to be compatible with the standard practice in digital asset management, i.e. store the metadata inside the digital asset like any other digital asset.

P.S.: I know we can still post on a closed issue but, by default, Redmine only presents open issues, and in general, people won't dig into closed issues to see if some should still be discussed. By closing the issue, you basically limit the discussion to you and the author, and ensure no one else will be involved in the conversation. Beside, "No way in hell" is not really an invite for a reasonable discussion.

#6 Updated by Roman Lebedev 6 months ago

Vincent Fregeac wrote:

"darktable edits your images non-destructively all the way through its pipeline. Your original image is never modified!" dixit Darktable website itself. "Non-destructive" editing is not a concept invented by Darktable, and it always meant parametric editing, in opposition to direct pixel modification. Also, note that darktable website present "non-destructive" editing as not modifying the image, it doesn't talk about the file containing the image. 32bit-float computation was implemented for another reason: to maintain fine nuances of tone and color throughout the editing and avoid losing information until the final image is converted back to 16bits; that's a different concept than non-destructive editing. But the important point is, none of these concepts are about the file, they are about not modifying the image contained in the file.

When it comes to managing the RAW files, Digital Asset Management is not a software, it's an entire industry with dozens of entreprise-grade solutions plus a few hundreds of smaller, specialized solutions. And the requirement of DAM solutions to have metadata saved inside the file is not a limitation, it's one of the basic principles of digital asset management: informations outside the digital asset will be lost at one point; the only way to ensure that metadata will not be lost is to incorporate the metadata in the file itself. That's how metadata are saved in all digital assets, whether they are text documents, audio files, video files, as well as almost every image format except some recent, proprietary, RAW formats. The only reason the concept of sidecars was created is because, initially, there was no tool to save metadata inside proprietary RAW formats as it is done with every other format of every other type of document. Sidecars were and still are a temporary patch, not a long term solution. Even Darktable acknowledges that since, when exporting images in another format than the proprietary RAW format, it saves metadata as they should: inside the image file.

Now, you can decide that Darktable will do things differently than the rest of the industry, but you can't expect an entire DAM industry to adapt to darktable. The only thing you will achieve is to make darktable incompatible with everything else and living in its own silo. I don't think that was the intent of the developers who created darktable 8 years ago.

Now, if exiv2 is still not reliable today, maybe it would be time to consider using exiftool instead. According to exiv2 wiki itself: "Today Exiftool is amazingly complete, it reads and writes metadata, supports more image formats, more types of metadata and more Makernotes than Exiv2 and Andreas considers it the reference implementation for image metadata." So even the main developer of Exiv2 basically recommends to use exiftool rather than exiv2, except for applications where you need to read a huge volume of metadata in one batch. See for your self: http://dev.exiv2.org/projects/exiv2/wiki/How_does_Exiv2_compare_to_Exiftool. So, for the typical use of metadata in darktable, i.e. importing metadata from a few hundred images per batch, or modifying metadata of a few dozen image together, the "slow" exiftool will still be faster than the fastest human fingers selecting or typing the metadata. And, yes, exiftool, although written in Perl, does have a C/C++ interface.

As for having backup before modifying a RAW file, well, even using a "non-destructive" editor who doesn't even touch the RAW file without a backup would be insane. Beside, every tool I have used who save metadata inside RAW file create a backup file by default (generally using an extension like .xxx_original). Since I prefer to manage my backups outside of the image tools, I deactivate that option but, considering a lot of people don't even backup their documents, having this option activated by default is still a good practice.

As I said, time have passed since the early day of exif editing, i.e.:
- saving metadata in RAW files is not the gamble it was before;
- exiv2 is apparently not the preferred solution anymore (except huge import of metadata), even according to exiv2 developers.

So, maybe it's time to reconsider the initial assumption that modifying a RAW file has no benefit and is too risky anyway. The environment images live in has evolved in the past 10 years, as well as the tool used to handle the metadata in RAW files.

So, in other words, you are choosing to completely disregard the words of a developer of another software that //does// write sidecars into raws https://bugs.kde.org/show_bug.cgi?id=387351
AND every bugreport that darktable got due to that digikam's switch, and exiftool fiddling?
All that still actively destroys raw files, even nowadays.

I am not saying darktable should only save metadata inside RAW files, I understand that some people are still paranoid about modifying RAW files: saving metadata and tags inside RAW files should only be an option, not even activated by default, even with a big disclaimer if you want. But darktable should not live in its own silo, disregarding any notion of compatibility with the rest of the tools used in an image workflow. At least because that was not the intent of the original developers: darktable was supposed to be a FOSS alternative, aimed at enthousiasts and professionals, taking into account and implementing the current best/standard practice in an image workflow. So, except if darktable now targets beginner photographers, it should take into consideration the standard practice in modern digital image management. And one of the changes in the best practice of digital image management is to consider RAW images as digital assets, not only the developed images. But it requires RAW files to be compatible with the standard practice in digital asset management, i.e. store the metadata inside the digital asset like any other digital asset.

P.S.: I know we can still post on a closed issue but, by default, Redmine only presents open issues, and in general, people won't dig into closed issues to see if some should still be discussed. By closing the issue, you basically limit the discussion to you and the author, and ensure no one else will be involved in the conversation. Beside, "No way in hell" is not really an invite for a reasonable discussion.

#7 Updated by Vincent Fregeac 6 months ago

I don't disregard the bugs of one implementation of exiftool, I just consider it as what it is: one failed implementation who, of course, gets all the attention, because people don't talk about things that just work.

But, beside digiKam failed implementation, there are others who just work, and have been working reliably for years, like XnView, GeoSetter, Lightroom, Bridge. Some limit the write feature to a very limited set of RAW formats, some simply create backups of RAW files by default because the developer doesn't have the time to test the reliability of the write feature for every forma, but they do work, and nobody talk about them because they just work.

Darktable's developers already have enough on their plate without becoming exiftool testers so I think the implementation of this feature should just come with a warning, in the form of dialog box saying that it should be use with caution, after doing your due diligence, because the support for some formats is still in beta, then the rest is the responsibility of the user. And, if you want to stay on the safe side, just create a copy of the original file, with a .xxx_original or .xxx_backup extension like XnView or GeoSetter does, preferably with the option not to create these backup files for those who are responsible grown-ups, capable of handling their backup duty themselves.

So, yes, digiKam implementation was and still is a failure and everybody keep talking about it and only about that one, and yes, the support to write in some of the RAW formats is still in beta and not ready for production, but darktable users are adults in general, able to make their own, responsible, decisions, they don't need to be fenced away from anything potentially unsafe. Darktable is not a school bus.

And for those who still don't do backups and act carelessly with their images, well, you learn by your mistakes, but you will never learn if you are fenced in the safe playground of a kindergarten. And don't treat everyone as irresponsible kids just because some still are.

#8 Updated by Vincent Fregeac 6 months ago

FYI: https://www.loc.gov/preservation/digital/formats/sustain/sustain.shtml#self

The Library of Congress has a better explanation than I do about "self-documentation" and why it's preferable.

Also available in: Atom PDF