Project

General

Profile

Bug #8651

Reviewing CR2 in lightable without using emb. jpg makes memory usage grows infinitely

Added by Emmanuel Lacour almost 8 years ago. Updated almost 7 years ago.

Status:
Closed: invalid
Priority:
Low
Assignee:
-
Category:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Affected Version:
1.0.4
System:
unknown
bitness:
64-bit
hardware architecture:
amd64/x86

Description

When reviewing a large folder of CR2 (~ 1000 files), without use of embedded jpg, the memory usage grows and never reduce. On a 4GB system (Intel 64bits) I went to near 90% of memory usage :(

Ask me if you need more details!

History

#1 Updated by Simon Spannagel almost 8 years ago

Hi,

you might wanna post the output of

darktable -d memory

and

darktable -d cache

to give us some more informations. Does thumbnail creation works with "using embedded jpegs" turned on?

#2 Updated by Simon Spannagel almost 8 years ago

  • Target version changed from 1.0.3 to Candidate for next patch release

#3 Updated by Simon Spannagel over 7 years ago

  • Affected Version set to 1.0.4
  • % Done changed from 0 to 20
  • Status changed from New to Incomplete

We need more information in order to be able to reproduce/fix this problem.

#4 Updated by Emmanuel Lacour over 7 years ago

About embedded jpegs, yes it works and does not seems to eat memory.

I just did a new test with latest release (1.0.5), I openned a collection with 998 CR2. Then zoomed out on the light table, memory used was near 11% in top, then I scrolled on the light table to see more thumbnails, memory grows to around, 28%, then again 48%, then 52% and it seems to stabilize here.

Near the beginning of the process, -dmemory gave:

[memory] after cache initialization
[memory] max address space (vmpeak): 481812 kB
[memory] cur address space (vmsize): 481812 kB
[memory] max used memory (vmhwm ): 146480 kB
[memory] cur used memory (vmrss ): 146480 kB

now it is:

[memory] before pixelpipe process
[memory] max address space (vmpeak): 6715636 kB
[memory] cur address space (vmsize): 6181848 kB
[memory] max used memory (vmhwm ): 4769764 kB
[memory] cur used memory (vmrss ): 4149164 kB

I'm going to test with -dcache.

#5 Updated by Emmanuel Lacour over 7 years ago

-d cache at the beginning (DT uses 10% of mem):

[mipmap_cache] level 0 fill 88.91/141.27 MB (62.93% in 1160/2048 buffers)
[mipmap_cache] level 1 fill 7.05/141.25 MB (4.99% in 23/512 buffers)
[mipmap_cache] level 2 fill 112.80/141.25 MB (79.86% in 92/128 buffers)
[mipmap_cache] level 3 fill 0.00/141.24 MB (0.00% in 0/32 buffers)
[mipmap_cache] level 4 fill 0.00/141.24 MB (0.00% in 0/8 buffers)
[mipmap_cache] level [full] fill 3/3 slots (100.00% in 3/16 buffers)

when DT is 30% of mem:

[mipmap_cache] level 0 fill 113.05/141.27 MB (80.02% in 1475/2048 buffers)
[mipmap_cache] level 1 fill 7.05/141.25 MB (4.99% in 23/512 buffers)
[mipmap_cache] level 2 fill 112.80/141.25 MB (79.86% in 92/128 buffers)
[mipmap_cache] level 3 fill 0.00/141.24 MB (0.00% in 0/32 buffers)
[mipmap_cache] level 4 fill 0.00/141.24 MB (0.00% in 0/8 buffers)
[mipmap_cache] level [full] fill 3/3 slots (100.00% in 3/16 buffers)

#6 Updated by Emmanuel Lacour over 7 years ago

And my DT settings:

- don't use jpeg embedded thumbnails
- mipmap size of 1Go
- 2 execution childs
- 2 concurrent exports
- 5Go memory limit for tile generation
- 256Mo for on tile cache
- no opencl
- nolittlecms2
- high resampling on

And my computer:

- AMD Phenom(tm) II X4 965 Processor
- 8Go RAM

#7 Updated by Tobias Ellinghaus over 7 years ago

So, do you see a memory leak or is it just a slow filling of the cache which stabilizes at some point? I can't see any memleak on my system when trying this.

#8 Updated by Alexey Akulov over 7 years ago

Had such memory issue when open a dir with over 500 jpg/ARW files, miniature number was set to default 89... Set it to 8900, now eats less memory...

[memory] before pixelpipe process
[memory] max address space (vmpeak):     3918836 kB
[memory] cur address space (vmsize):     3905196 kB
[memory] max used memory   (vmhwm ):     1628536 kB
[memory] cur used memory   (vmrss ):     1581808 kB
[memory] before pixelpipe process
[memory] max address space (vmpeak):     4372780 kB
[memory] cur address space (vmsize):     4365504 kB
[memory] max used memory   (vmhwm ):     1856752 kB
[memory] cur used memory   (vmrss ):     1856752 kB
[memory] before pixelpipe process
[memory] max address space (vmpeak):     4372780 kB
[memory] cur address space (vmsize):     4032956 kB
[memory] max used memory   (vmhwm ):     2180388 kB
[memory] cur used memory   (vmrss ):     2044324 kB
Killed

#9 Updated by Simon Spannagel almost 7 years ago

  • System set to unknown
  • % Done changed from 20 to 0
  • Status changed from Incomplete to Closed: invalid

I consider this invalid, I could not find any issues importting/viewing folders with several thousand CR2s.

Also available in: Atom PDF

Go to top