Feature #9836

Lua: thumbnail support

Added by Steve Pomeroy about 5 years ago. Updated over 4 years ago.

Target version:
Start date:
Due date:
% Done:


Affected Version:
git development version
hardware architecture:


I'm developing a Lua script that deals with offline image management (git-annex integration) and would like to have more control of the thumbnails from Lua. In particular, I'd like to be able to refresh thumbnails (force a regeneration of them from the source file), so that when an image's backing file has been re-copied to the working folder, the plugin can refresh it.

Additionally, as this script is working with a bunch of offline photos, it would be ideal the thumbnails could be more permenantly cached so that I don't get the dreaded skull if the cache expires. Perhaps there could be a flag settable on an image that tells darktable to not expire its thumbnail?


#1 Updated by Jérémy Rosen about 5 years ago

IRC conversation on the subject (so it doesn't get lost)

(09:35:26) boucman_work: hanatos: if you have a minute... quick question
(09:35:53) boucman_work: is there a way to control what thumbnails are kept in cache and what thumbnails are dropped ? is there a way to force the regeneration of a thumbnail ?
(09:36:13) boucman_work: with lua, some people want to manipulate that...
(09:51:30) hanatos: boucman_work: minutes are scarce this week again :(
(09:51:35) hanatos: i don't think there is such a mechanism
(09:51:38) hanatos: for any of your questions :)
(09:51:47) hanatos: you can call dt_mipmap_cache_prefetch()
(09:51:53) hanatos: to force loading of a specific thumb
(09:51:59) hanatos: dropping is done on a lru basis
(09:52:13) boucman_work: and regenerating ?
(09:52:32) hanatos: regenerate?
(09:52:36) hanatos: thumbs are either in cache
(09:52:40) hanatos: or loaded on demand
(09:52:50) hanatos: meaning you try to access them and if they are not there
(09:52:54) hanatos: you get one with lower res
(09:52:58) hanatos: all the way to mip0
(09:53:06) hanatos: while prefetching is called for the one you requested
(09:53:18) hanatos: so you could script a create-all-thumbs thing
(09:53:26) hanatos: by iterating over dt_mipmap_cache_prefetch()
(09:53:38) hanatos: given a large enough cache of course
(09:53:43) hanatos: or else they will evict older thumbs
(09:56:08) boucman_work: mkay...
(09:58:05) boucman_work: to be a bit more specific i'm looking at
(10:32:25) hanatos: boucman_work: 1) regenerate is possible 2) keeping thumbs makes it not a cache but grow indefinitely. you can just set insane memory bounds, it'll grow gradually
(10:32:35) hanatos: (which i find is i dumb idea)
(10:37:34) boucman_work: it makes sense to keep thumbnails of offline images, so you know which one to request over the network, but it's a different use-case than the thumbnail cache
(10:38:04) boucman_work: question is : should I (ab)use the mipmap cache or is there another way to do that (store a cold cache somewhere that is not LRU)
(11:11:43) hanatos: boucman_work: it might make sense to store mip4 thumbnails on disk, next to the image or in xmp
(11:11:55) hanatos: certainly makes no sense to store it in cache format
(11:12:01) hanatos: you would never get it back
(11:12:15) hanatos: not sure how easy it is to keep that stuff in sync
(11:12:28) hanatos: if the cache is the only one writing to it it might be simple
(11:12:37) hanatos: evict thumb from cache => write it to disk
(11:12:43) hanatos: do the same on shutdown for all the thumbs
(11:12:58) hanatos: load it again: load from disk first, then recreate
(11:13:16) hanatos: maybe that'll make thumb loading from cache so fast that users wouldn't notice it any more
(11:15:15) boucman_work: so next to xxx.raw and xxx.xmp add xxx.mip4 for thumbnails ?
(11:16:00) boucman_work: that's way more complicated than just blocking the eviction, but that makes sense
(11:17:11) pmjdebruijn: is that mip4 just a JPG?
(11:17:51) pmjdebruijn: or something special?
(11:17:54) boucman_work: that was my next question :P
(11:18:10) boucman_work: we can't call it .jpg though, that would screw the import logic
(11:18:17) pmjdebruijn: true
(11:22:46) boucman_work: so add a "keep mipmap" flag to images, if true then save to disk on eviction, and load from disk instead of recreating if it's up to date and available on disk
(11:23:10) boucman_work: or no flag and just always save to disk when generating
(11:23:18) boucman_work: or save in the xmp :P

#2 Updated by Steve Pomeroy about 5 years ago

I like the notion of optional sidecar thumbnails. Also: evicting from cache to disk is particularly clever, as that should prevent dupes on disk, although it may be a bit confusing to only see the sidecar thumbnails for some images. Maybe just have it so that every time the cache is written to disk, an option lets users choose a dual write to the sidecar thumbnail as well (or instead)?

As far as naming a sidecar thumbnail, keeping it with the original is good (though IMO not necessary, as they are automatically generated shadows of the content and can be recreated at-will. It is unlikely, though possible in my particular usecase (git-annex), that someone would move offline files around on disk). Perhaps use standard *nix convention and name them with a "~" at the end or start the name with a "."?

#3 Updated by Steve Pomeroy about 5 years ago

I forgot to mention: if possible, for my use case it'd be better if it was not stored in the xmp file as I check that into git to manage styling revisions.

#4 Updated by Tobias Ellinghaus about 5 years ago

No worries, the XMP files will definitely not contain any thumbnails.

#6 Updated by Jérémy Rosen over 4 years ago

  • bitness set to 64-bit
  • Affected Version set to git development version
  • System set to all

I recently added an API to evict an image from the cache. That would partially solve your problem (no pinning of mipmap yet)

Also available in: Atom PDF