Project

General

Profile

Bug #10167

Slow when using it with a lot of Images

Added by Roland Schuller over 4 years ago. Updated almost 4 years ago.

Status:
Incomplete
Priority:
Low
Assignee:
-
Category:
Lighttable
Start date:
10/16/2014
Due date:
% Done:

20%

Affected Version:
1.4.2
System:
Debian
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Hi!

I'm using Darktable for processing a lot of Images (i mean at least 1000 per Directory)
It takes some time to import, but darktable does not create thumbnails by itselve.
I have to slowly scroll down to generate the thumbs an to get an quit overview of my images.

When i close Darktable and reopen it -> the sae procedure. This is a bit annoying.
I'd like to have the thumbnails of the Images saved so they wont be recalculated every time but only on change.

I know it is a lot of Data (~31 MB per NEF) (last project 1037 files) ... it take time to process. But it should process them alltogether so i can quick scroll though them.

System Information:
Core i7-3820 CPU @ 3.60GHz
Ram 16Gb
OpenCL : GeForce GT 740
Settings: (simple translated from German)
MIP-Map-Cache: 4069
Background threads: 16
MemoryLimit: 0
Memory Ammount: 96

History

#1 Updated by Tobias Ellinghaus over 4 years ago

Try to set the number of background threads to 2. That might speed it up already.

#2 Updated by Roland Schuller over 4 years ago

Changed from 16 to 2. Cant tell any difference.
In my systemmonitor i see that ist is working, but when all images i see right now have ben calculated it stops completly. When i then scroll down the List i see empty images containers (gray) and the system is calculating again.
I need more than 10 minutes this way to get all thumbs to get a complete overview.
I like this tool and I dont want to work with something other, but this is annoying.

#3 Updated by Tobias Ellinghaus over 4 years ago

  • Status changed from New to Closed: invalid

Ah, now I get it. That is intended behaviour, only those thumbnails that are needed are actually generated. Plus some more for prefetching. It might eventually be possible in 1.6 to have a lua script trigger thumbnail generation. Maybe.

#4 Updated by Tobias Ellinghaus over 4 years ago

  • Status changed from Closed: invalid to Incomplete
  • % Done changed from 0 to 20

Oh wait, do you mean that thumbs are gone again after restarting dt?

#5 Updated by Roland Schuller over 4 years ago

When i reload an older Project I only see some of them.
Sometimes they are all gone.
But my last Project keeps all thumbs. Cant find any clue how and why.
Where are the thumbs stored?

#6 Updated by Jérémy Rosen over 4 years ago

in 1.6 lua will allow you to DROP thumbnails, not force regeneration

the result of discussions on the subject was that there were no real cases where lua would do a better job than DT at knowing what thumbnails to generate, so it was deemed better not to provide self-hanging rope in lua.

If you have so many issues with thumbnails you probably want to increase your mipmap cache size so more images stay in there... (Maybe DT should be more aggressive at keeping low res previews, only droping those as a last resort and dropping higher res thumbnails when possible, that's another debate...

#7 Updated by Tobias Ellinghaus over 4 years ago

He has 4gb compressed thumbs already. That should be plenty.

#8 Updated by Pascal de Bruijn over 4 years ago

"When i reload an older Project I only see some of them."

What do you mean by "reload an older project", in Darktable's context that makes very little sense. Are you...

1. Reselecting an already imported filmroll via the 'collect images' plugin?
or
2. Reimporting an already imported directory again? Which effectively means you're forcing Darktable to reindex that Directory.

#9 Updated by Roland Schuller over 4 years ago

First of all its when i import a new Directory (completely new from my Cam) then I hve to manually scroll down , wait, scroll, wail, scroll,... till i get a overview of all Images. Its a big problem when I try to find a certain Part.

When I reopen a project via 'Bilder sammeln - collect Images' or 'kürzlich benutzte Sammlungen - recent used' sometime i see no thums at all, sometimes i see some thumbs, sometimes they are all there.

What might help. All my collextions are connected to my PC via samba share because they are on a Server (Gbit Network, server 2 Meters away)
Now I removed the DT database so it hase to reindex all Images. But the effect has not changed. (Had a lot of time to reindex all recently used Images)
My Imagecount (find | grep ".NEF"|wc -l) for 2014 gave me 22367 Images ind 48 Folders.

#10 Updated by Pascal de Bruijn over 4 years ago

Can you reproduce this issue when your files are on your local filesystem?

#11 Updated by Roland Schuller over 4 years ago

I only have one Folder on my local Disk (My Server has 1.5T Space)
The System is much more stable here. When I scrolled down, they wehre all here. When I scrolled up again i found that the first ~15 where missing (Orderd by colour)
The i went back to a Projekt I worked with yesterday. (On my Server) -> No thumbs at all.
Yesterday I had the effect, that I previewed all New Images in Lighttable (Zoom Level 1) and removed the bad ones (pressing '0'). The i got back to Zoom level 3 -> no Thumbs. WTF.

#12 Updated by Martin Milinovsky almost 4 years ago

I see the same effect: when opening an already imported folder dt generates the thumbnails very slowly and only when scrolling down - as decribed from Roland Schuller in #9. I also have the images on a fileserver which I access via Samba over wlan. If I make "local copies" the performance is a lot better.

#13 Updated by Pedro Côrte-Real almost 4 years ago

  • Target version set to Candidate for next major release

At a guess what's happening here is the cache size is not large enough for the number of images selected at the resolution that's been defined. This should be fixed in current master (will be 2.0 when released). The new cache saves thumbnails as jpg to disk so once you've generated a thumbnail once loading it again should be much faster. It also selects the image sizes automatically so there's no longer much need to tweak the cache settings to adapt to different systems and monitor sizes.

As for Martin's situation that's to be expected on first open. Samba over wlan will be much slower than local files, there's nothing darktable can do about that. But since we will now save jpg thumbnails locally reopening old folders should work much better.

Also available in: Atom PDF