Project

General

Profile

Bug #10479

Darktable constantly regenerating mipmaps

Added by Francisco Cribari about 4 years ago. Updated over 3 years ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
Lighttable
Start date:
05/22/2015
Due date:
% Done:

100%

Affected Version:
1.6.6
System:
all
bitness:
64-bit
hardware architecture:
amd64/x86

Description

I am running Darktable 1.6.6 on an Ubuntu notebook (15.04, 64 bt). When I am in lighttable DT seems to be always generating again and again the thumbnails and I move down or up. My images are in an external HD (ext4 filesystem) connected through an USB 3.0 port. I am attaching the output of "darktable -d all". My library.db has only 5.2 MB.

cribari@darwin4:~/.config/darktable$ ls lh
total 5,4M
-rw-r--r-
1 cribari cribari 8,5K Mai 19 20:35 darktable.gtkrc
rw-rw-r- 1 cribari cribari 17K Mai 22 11:00 darktablerc
rw-r--r- 1 cribari cribari 75K Mai 19 20:35 keyboardrc
rw-rw-r- 1 cribari cribari 75K Mai 22 10:58 keyboardrc_default
rw-r--r- 1 cribari cribari 5,2M Mai 22 10:59 library.db
drwxrwxr-x 2 cribari cribari 4,0K Mai 21 17:19 styles
drwxrwxr-x 2 cribari cribari 4,0K Mai 21 06:36 watermarks

History

#1 Updated by Roman Lebedev about 4 years ago

What is the value of "memory in megabytes to use for mipmap cache" and how many images in the collection?
You might fix it by increasing that setting.

Since in master we store thumbs as separate JPG's, the issue will almost fully go away in 2.0

#2 Updated by Francisco Cribari about 4 years ago

The value I have in "memory in megabytes to use for mipmap cache" is 512. The collection has 583 images.

Roman Lebedev wrote:

What is the value of "memory in megabytes to use for mipmap cache" and how many images in the collection?
You might fix it by increasing that setting.

Since in master we store thumbs as separate JPG's, the issue will almost fully go away in 2.0

#3 Updated by Francisco Cribari about 4 years ago

Also... I increased the value in memory in megabytes to use for mipmap cache to 1024. The problem remains.

Francisco Cribari wrote:

The value I have in "memory in megabytes to use for mipmap cache" is 512. The collection has 583 images.

Roman Lebedev wrote:

What is the value of "memory in megabytes to use for mipmap cache" and how many images in the collection?
You might fix it by increasing that setting.

Since in master we store thumbs as separate JPG's, the issue will almost fully go away in 2.0

#4 Updated by Francisco Cribari about 4 years ago

There were reports of similar behavior in the darktable users mailing list. For instance:

Date: Sat, 09 May 2015 12:51:54 +0200
From: Bernhard <XXXXXXX@XXXXXXXX>
Subject: Re: [Darktable-users] Darktable hanging on Linux
To:
Message-ID: <XXXXXXXXXXXX>
Content-Type: text/plain; charset="windows-1252"

Seems it is the same that happens to me:
changing from one big collection to another causes dt to rebuild the
thumbnails.
Trying to open a picture in darkroom mode before all thumbnails have
been recreated gives a big chance that dt will be unrespondable and I
have to close and restart.

File system is ext4 for / and ext3 for /home.

I hope this will be resolved by the permanent thumbnail cache planned
vor V2.

Bernhard

Scott schrieb am 08.05.2015 um 02:46:

I have this issue as well. Not all the time, but often if I import a
set of images and select one to view in darkroom before they are all
loaded, DT freezes. I'm using ext4 file system. Sometimes if I leave
it alone long enough, DT responds again....but I've left it for 5+
minutes before with no response. Like Oliver, I also wipe my library
and just import what I'm working on, so I'm sure it is not a library
issue. Ubuntu 14.04 and I've seen this issue with all versions of DT

-Scott

#5 Updated by Francisco Cribari about 4 years ago

That also happens to me: "Trying to open a picture in darkroom mode before all thumbnails have
been recreated gives a big chance that dt will be unrespondable and I have to close and restart."

Francisco Cribari wrote:

There were reports of similar behavior in the darktable users mailing list. For instance:

Date: Sat, 09 May 2015 12:51:54 +0200
From: Bernhard <XXXXXX@XXXXXX>
Subject: Re: [Darktable-users] Darktable hanging on Linux
To:
Message-ID: <XXXXXXX@XXXXXXX>
Content-Type: text/plain; charset="windows-1252"

Seems it is the same that happens to me:
changing from one big collection to another causes dt to rebuild the
thumbnails.
Trying to open a picture in darkroom mode before all thumbnails have
been recreated gives a big chance that dt will be unrespondable and I
have to close and restart.

File system is ext4 for / and ext3 for /home.

I hope this will be resolved by the permanent thumbnail cache planned
vor V2.

Bernhard

Scott schrieb am 08.05.2015 um 02:46:

I have this issue as well. Not all the time, but often if I import a
set of images and select one to view in darkroom before they are all
loaded, DT freezes. I'm using ext4 file system. Sometimes if I leave
it alone long enough, DT responds again....but I've left it for 5+
minutes before with no response. Like Oliver, I also wipe my library
and just import what I'm working on, so I'm sure it is not a library
issue. Ubuntu 14.04 and I've seen this issue with all versions of DT

-Scott

#6 Updated by Francisco Cribari about 4 years ago

The problem happens with all my collections, even with those that only have a few RAW files and no JPG files.

#7 Updated by Francisco Cribari about 4 years ago

(Mr Bernhard asked me to anonymize the e-mail-addresses in the messages from the Darktable users list I quoted. Unfortunately, I was unable to edit the comments above where I quote such messages. I thus ask that anyone with site privileges to edit messages to do so. Thank you and apologies to Mr Bernhard.)

#8 Updated by Oliver Bedford about 4 years ago

Francisco Cribari wrote:

There were reports of similar behavior in the darktable users mailing list.

I've had the same trouble Fracisco reported. Even increasing the mipmap cache to 4096 did not help.

What in effect does seem to help, was purging my complete dt configuration.
Of course this is not a solution and may not even be a work-around for everybody.

Currently with 1.6.6. I'm not able to provoke a hang. library.db is 9.8M.

Regards,
Oliver

#9 Updated by John Morris about 4 years ago

I have had this trouble with the last several versions, including 1.6.7.

John.

#10 Updated by Francisco Cribari almost 4 years ago

I am now running Darktable 1.6.7 on a Fedora 22 notebook (Intel i7, 8 GB RAM). I am having the same problem. Darktable seems to be constantly regenerating the mipmaps and it crashes when I try to enter darkroom before it regenerates all mipmaps. I increased the memory to use for mipmap cache to 1024 but the problem remains. It is quite annoying to see Darktable crash over a dozen times in less than one hour or so... Suggestions are welcome. Thank you.

John Morris wrote:

I have had this trouble with the last several versions, including 1.6.7.

John.

#11 Updated by Francisco Cribari almost 4 years ago

I started Darktable with darktable -d all . The output (up to the crash) is available at

https://dl.dropboxusercontent.com/u/2171814/out-2.txt

Is there anything I could try?

Francisco Cribari wrote:

I am now running Darktable 1.6.7 on a Fedora 22 notebook (Intel i7, 8 GB RAM). I am having the same problem. Darktable seems to be constantly regenerating the mipmaps and it crashes when I try to enter darkroom before it regenerates all mipmaps. I increased the memory to use for mipmap cache to 1024 but the problem remains. It is quite annoying to see Darktable crash over a dozen times in less than one hour or so... Suggestions are welcome. Thank you.

#12 Updated by Stéphane Gourichon almost 4 years ago

Similar problem

Similar problem observed here for months, probably more than a year, including with 1.6.7 and older non-released compiled from git-based sources.

Hardware: i7 quad-core, 6GB RAM, machine load (CPU and memory) is light, hard drive can sustain tens of megabytes of bandwidth, observed on 64-bit Ubuntu 14.04 and 14.10 (both recently).

How to reproduce

  • find a directory where all pictures (even subdirs) are already known to darktable
  • open a shell in that directory, run "darktable ." or "darktable *.NEF"
  • window opens, thumbnails appear reasonably quickly
  • for a long time, a progress bar at bottom left progresses (say, maybe a minute if directory contains more than a hundred of pictures)
  • double click on any thumbnail before progress bar completes and disappears
Observed:
  • darktable consumes CPU continuously and window content is frozen, no change, even after many minutes. I have to eventually kill darktable.
Expected:
  • darktable shows picture and allows edit, regardless of whether the thumbnail was double-clicked before or after the progress bar completed
Reproducible:
  • probably always (cannot test right now)
  • seen many times to the point I have the habit to just wait for the progress bar to complete

Additional information

Should we consider this a different bug ? File it separately ?

I often use darktable remotely through an Ethernet cable, Gigabit speed, to a very similar machine via SSH (with DISPLAY forward via ssh option -X). darktable nearly always works very well in that setup, only slightly slowed down interaction despite the bandwidth needed for a 2560x1440 screen. I observed a few times in this setup a situation where the thumbnails were constantly flashing on screen (maybe one at a time, but every few seconds each and IIRC alternating between a very rough appearance and a nicer finer rendering).

Question

Francesco, I see that the bug "Description" field does not mention crashes, while your update and e-mail today do. Can you compare your observations with mine and make the "Description" reasonably complete and up-to-date ? It may help figure out if it's the same bug.

#13 Updated by Tom Vijlbrief almost 4 years ago

I have seen this issue (the hang) on my i7 (8 hyperthreaded cores) but never on my dual core AMD. Looks like a race condition which is more likely to occur on machines with many cores?

Edit: I got a crash with the current master and threads set to 3 on my dual core:

~/opt/darktable/bin/darktable .
[New LWP 8472]
[New LWP 8469]
[New LWP 8466]
[New LWP 8465]
[New LWP 8452]
[New LWP 8451]
[New LWP 8446]
[New LWP 8445]
[New LWP 8444]
[New LWP 8443]
[New LWP 8442]
[New LWP 8441]
[New LWP 8440]
[New LWP 8439]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
malloc_consolidate (av=av@entry=0x7fb26d7eac00 <main_arena>) at malloc.c:4150
4150 malloc.c: No such file or directory.
backtrace written to /tmp/darktable_bt_9DKM0X.txt
Segmentation fault

http://v7f.eu/public/darktable/darktable_bt_9DKM0X.txt

#14 Updated by Tom Vijlbrief almost 4 years ago

Probably unrelated but when I pass a directory to a DT build with debugging (trying to reproduce this issue) I get:

darktable: /home/tom/src/darktable/src/lua/init.c:155: dt_lua_init: Assertion `lua_gettop(L) == 0' failed.
Aborted

#15 Updated by Tobias Ellinghaus almost 4 years ago

Tom Vijlbrief wrote:

a DT build with debugging

What do you mean? Passing "-d" to get some output, or compiling as -DCMAKE_BUILD_TYPE=Debug?

#16 Updated by Tom Vijlbrief almost 4 years ago

./build.sh --buildtype Debug

#17 Updated by Tobias Ellinghaus almost 4 years ago

That should be impossible, we explicitly check for that in CMake and fail when compiling as Debug. I will have a look later.

#18 Updated by Francisco Cribari almost 4 years ago

I am still having this problem (Darktable 1.6.7, Fedora 22). Is there anything I should try? Thank you.

#19 Updated by Francisco Cribari almost 4 years ago

This video shows the problem I've been having:

https://www.youtube.com/watch?v=mJeykGhacXk

#20 Updated by Pedro Côrte-Real almost 4 years ago

What I see in the video is pretty much expected behavior. With the 1.6 cache ~500MB won't fit very many thumbnails so as you scroll the older ones get evicted from cache and have to be regenerated later. With 2.0 the thumbnails will be saved to disk instead so you should get much faster thumbnail performance.

#21 Updated by Pedro Côrte-Real over 3 years ago

  • % Done changed from 0 to 100
  • Status changed from New to Fixed
  • Target version changed from Candidate for next patch release to Candidate for next major release
  • System changed from Ubuntu to all

The new cache that will come out in 2.0 does not suffer from this issue anymore. The 1.6 series won't get any more fixes for this though. We're in feature freeze for 2.0 so darktable compiled from master should be ok to test by now (you can't go back to 1.6 after that though).

Also available in: Atom PDF