Bug #9621

darktable deletes all but the first sidecar files when images are removed from a collection

Added by Ulrich Pegelow over 6 years ago. Updated over 6 years ago.

Start date:
Due date:
% Done:


Estimated time:
Affected Version:
git development version
hardware architecture:


Here is how to reproduce:

1) take an image into a filmroll and make some adjustments
2) produce a duplicate of that image with distinguishable settings (eg monochrome)
3) remove both images from your collection
4) re-import the filmroll

=> result: only one image (the first) appears in the filmroll, the second one is silently ignored

=> what should happen: all history stacks should be recognized and lead to duplicates


#1 Updated by Ulrich Pegelow over 6 years ago

  • Priority changed from Medium to Critical

Re-checked and problem is even worse. When removing an image with duplicates from a collection all but one sidecar files are physically deleted and not recoverable :(

Needs an urgent fix.

#2 Updated by Ulrich Pegelow over 6 years ago

  • Subject changed from re-importing images does not recognize multiple history stacks to darktable deletes all but the first sidecar files when images are removed from a collection

Seems like a long-standing issue - maybe never worked correctly. I tried with HEAD~1000 and it has the same problem.

Looking into the code in control/jobs/control_jobs.c I wonder how it's supposed to work. In dt_control_remove_images_job_run()
we first remove each of the selected images: "while(t) { ... dt_image_remove(imgid) ... }" and then
loop over the list of images to resynch the xmp's: "while(list) { ... dt_image_synch_all_xmp(imgname) ... }". To my understanding that
would not function as removing an image also removes all db entries: no chance to reconstruct xmp's ...

Wonder if we could just change the sequence: first re-synch all xmp's and then do the actual image removal.

#3 Updated by Ulrich Pegelow over 6 years ago

Update: the situation is more difficult than first thought as this is not an obvious bug but merely a "feature" which might be difficult to change.

When we have several duplicates of an image in a filmroll they all are represented by individual XMP files - the first one with <filename>.<ext>.xmp, the
others with <filename>_nn.<ext>.xmp where nn represents consecutive two-digit numbers starting with 01. Currently darktable makes sure that those numbers are re-arranged and stay consecutive even if one of the duplicates is removed. This implies that some XMP files get deleted and others overwritten. There is no option to remove one of the duplicates from the database but keep its XMP file.

As a consequence, if all images are removed only the base XMP <filename>.<ext>.xmp can stay all other XMP's will be deleted.

That's at least unintuitive. My thought was that I can safely remove images from darktable, then backup the folder and be able to fully reconstruct my work by reimporting the backup.

We certainly need a warning in the usermanual!

#4 Updated by Tobias Ellinghaus over 6 years ago

  • System changed from openSUSE to all
  • % Done changed from 0 to 20
  • Status changed from New to Triaged

We don't need a warning but a fix. This is putting user's edits at risk. I'd rather remove this "feature" than losing my work.

#5 Updated by Ulrich Pegelow over 6 years ago

A fix would involve a changed numbering scheme for XMP's with a clear link between XMP-name and imgid.

My suggestion would be: <filename>_dtABC.<ext>.xmp

ABC represents the last three digits of imgid. (I'm open to have more digits but three should suffice in all practical cases).

The XMP file gets removed if we DELETE an image from a collection. The XMP file gets preserved if we REMOVE an image from a collection.
On re-importing a filmroll or image, darktable needs to look for all XMP's that follow that naming scheme. If it detects an XMP which does not
correspond to an imgid in the DB, a new duplicate is generated taking the parameters in that XMP; then that old XMP is removed and a new one generated
with the right name reflecting the new imgid in DB.

We also should fix another related issue. Currently darktable assumes that all XMP's that fit to <filename>_*.<ext>.xmp belong to an input image
<filename>.<ext>. This has some bad side effects. If we have two images in our film roll: brother.jpg and brother_and_sister.jpg and we have multiple
duplicates of brother.jpg, darktable will remove any brother_and_sister*.jpg.xmp that might exist as soon as the XMP's for brother.jpg need to get
updated :(

#6 Updated by Ulrich Pegelow over 6 years ago

  • % Done changed from 20 to 50
  • Assignee set to Ulrich Pegelow
  • Status changed from Triaged to In Progress

#7 Updated by Tobias Ellinghaus over 6 years ago

Another way to fix it is to have the _nn part of the xmp file in the db. That way we have no clashes with imgids, don't expose internal ids to the outside and keep the same filename when importing the duplicate.

#8 Updated by Ulrich Pegelow over 6 years ago

Main advantage of imgid: it's an autoincrement field (at least if the DB is not extremely old). As a consequence it remains unique which is difficult to guarantee for _nn. Suppose the case where a user has made a backup of his images at some point in time and continues work (duplicate, remove duplicates etc) in his working folder. At a later point he notices that he has deleted a duplicate he would like to recover from backup. With imgid it's clear for darktable which of the backup xmp's need to be re-read and which not.

#9 Updated by Ulrich Pegelow over 6 years ago

@Tobias, I follow your route and will keep track of versioning (incl. max_version) within the images table.

Short discussion about one special sidecar file, i.e. <filename>.<ext>.xmp. Currently we have this one for the first duplicate and filenames <filename>_nn.<ext>.xmp for all other duplicates where nn starts with 01. On each change currently the sidecar numbering is rearranged with the described problem.

In future I see two options:

a) <filename>.<ext>.xmp is a synonymon for <filename>_00.<ext>.xmp
In most cases this leaves things as user expects. If there is only one version then there is only <filename>.<ext>.xmp. However things can get complicated if a user has several duplicates and then DELETES the first (original) one. In that case there is no <filename>.<ext>.xmp but only the numbered ones which refer to the other versions that have been constructed. Users might find this strange and external programs might miss the "standard" sidecar file. On the positive side: there is always a direct relation from filename and version.

b) <filename>.<ext>.xmp is always a (hard) link to the lowest version number in the collection
In this case there is always a <filename>.<ext>.xmp and it typically links to <filename>_00.<ext>.xmp (We might decide to not generate the _00 sidecar if only one version exists and this version has number 0). In the above example deleting or removing the 0-version would just make <filename>.<ext>.xmp link to the next still existing version number. Keeping track of the links is a bit more complicated then and it's much more complicated to decide which version <filename>.<ext>.xmp relates to, when we want to re-import a filmroll.

Any feelings about the two options?

#10 Updated by Tobias Ellinghaus over 6 years ago

No links please, that will not work on some file systems. I don't know about network shares, but at least FAT variants will most probably not support that. And many people use them to be able to easily transfer data between computers. Links might also make trouble when archiving data.

#11 Updated by Ulrich Pegelow over 6 years ago

Yepp, I fully agree. Option (a) for me is the preferred one which I currently have started to implement.

#12 Updated by Ulrich Pegelow over 6 years ago

  • % Done changed from 50 to 100
  • Status changed from In Progress to Fixed

Also available in: Atom PDF

Go to top