Project

General

Profile

Bug #11118

SQLite issues. Was: darktable crashing on import

Added by David Wen Riccardi-Zhu about 2 years ago. Updated about 2 years ago.

Status:
Triaged
Priority:
Medium
Assignee:
-
Category:
-
Target version:
-
Start date:
08/19/2016
Due date:
% Done:

20%

Affected Version:
git master branch
System:
all
bitness:
64-bit
hardware architecture:
amd64/x86

Description

I am trying to import a large photo folder on an external drive -- ~95K photos, 3.5K folders, ~1.3TiB.

Darktable imports about 1/8 of the photos, then hangs.

I'm not sure how to provide any output/bug report from the program, but would be happy to do so.

Thanks for the great work you are doing.

darktable-dbg-sql.txt.tar.gz (303 KB) David Wen Riccardi-Zhu, 08/24/2016 03:25 AM

darktable-dbg-sql-20160825.txt.tar.gz - Attempt 2. (553 KB) David Wen Riccardi-Zhu, 08/25/2016 09:58 PM

History

#1 Updated by David Wen Riccardi-Zhu about 2 years ago

Ran from terminal, getting the following:

Eight times: [file].jpg.xmp: caught exiv2 exception '[xmp_write] failed to serialize xmp data'

Then:

sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): near "X": syntax error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): unrecognized token: ""
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): near "lect": syntax error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error

#2 Updated by Roman Lebedev about 2 years ago

IIRC the import time has an issue of quadratic growth, so i'd highly recommend importing only small chunks at a time (no more than 10k images at once)

#3 Updated by David Wen Riccardi-Zhu about 2 years ago

Roman Lebedev wrote:

IIRC the import time has an issue of quadratic growth, so i'd highly recommend importing only small chunks at a time (no more than 10k images at once)

Thanks! That seems to be the case. Imported a few albums, first one over 10K crashed again.

#4 Updated by Roman Lebedev about 2 years ago

  • Status changed from New to Incomplete
  • % Done changed from 0 to 20

If it crashes, it should create a backtrace in /tmp/darktable_bt_XXXXXX.txt

#5 Updated by David Wen Riccardi-Zhu about 2 years ago

Roman Lebedev wrote:

If it crashes, it should create a backtrace in /tmp/darktable_bt_XXXXXX.txt

Thanks -- sadly I'm not seeing anything in that directory. : \ Ubuntu 16.04.

#6 Updated by Roman Lebedev about 2 years ago

David Wen Riccardi-Zhu wrote:

Ran from terminal, getting the following:

Eight times: [file].jpg.xmp: caught exiv2 exception '[xmp_write] failed to serialize xmp data'

Then:

sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): near "X": syntax error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): unrecognized token: ""
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:314, function _update_collected_images(): near "lect": syntax error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:315, function _update_collected_images(): not an error
sqlite3 error: /build/darktable-3aW75U/darktable-2.0.5/src/views/lighttable.c:316, function _update_collected_images(): not an error

Hm, can you find out on approximately which files that happens?
Do they have strange filenames?
Do they have strange tags?
Maybe they contain special characters like '"[] and so on?

#7 Updated by Roman Lebedev about 2 years ago

Second thought: run darktable with -d sql, redirect output to file:
darktable -d sql 2>&1 > /tmp/darktable-dbg-sql.txt
And then upload that file.

Maybe that will help us...

#8 Updated by David Wen Riccardi-Zhu about 2 years ago

Roman Lebedev wrote:

Second thought: run darktable with -d sql, redirect output to file:
darktable -d sql 2>&1 > /tmp/darktable-dbg-sql.txt
And then upload that file.

Maybe that will help us...

Thanks, Roman! It's attached.

It's a bit hard to say about the actual filenames, but I don't think there should be anything out of the ordinary. My typical filename is, for example, dwrz_20160824_0000.dng.

The really strange thing is this: I just tried to import albums 2006, 2007, and 2008. 2006 crashed the last time I tried, this time it worked. 2007 did not work the first two times I tried, but it worked the third time. So far, 2008 has not been working. I think the albums have been around 10-12K images.

[By the way -- Lightroom is the only software that's making me keep Windows around. I still have to see whether the darktable setup is the right fit for me, but I am trying to make an effort to give it a shot. Bottom line is that I am very grateful for all your work and hope to continue to contribute bug reports (at least, and since I can't code yet). Many thanks to the darktable team.]

#9 Updated by David Wen Riccardi-Zhu about 2 years ago

Looks like the file did not attach.

#10 Updated by David Wen Riccardi-Zhu about 2 years ago

David Wen Riccardi-Zhu wrote:

Looks like the file did not attach.

One more time . . . (trying tar this time)

#11 Updated by Roman Lebedev about 2 years ago

David Wen Riccardi-Zhu wrote:

David Wen Riccardi-Zhu wrote:

Looks like the file did not attach.

One more time . . . (trying tar this time)

Hmm, i do not see any "sqlite3 error:" lines in there?
Perhaps i failed to provide you with the right line, please try darktable -d sql &> /tmp/darktable-dbg-sql.txt

#12 Updated by David Wen Riccardi-Zhu about 2 years ago

Attached! File is ~175mib uncompressed.

Roman Lebedev wrote:

David Wen Riccardi-Zhu wrote:

David Wen Riccardi-Zhu wrote:

Looks like the file did not attach.

One more time . . . (trying tar this time)

Hmm, i do not see any "sqlite3 error:" lines in there?
Perhaps i failed to provide you with the right line, please try darktable -d sql &> /tmp/darktable-dbg-sql.txt

#13 Updated by Roman Lebedev about 2 years ago

David Wen Riccardi-Zhu wrote:

Attached! File is ~175mib uncompressed.

File upload seems to be broken :(

Roman Lebedev wrote:

David Wen Riccardi-Zhu wrote:

David Wen Riccardi-Zhu wrote:

Looks like the file did not attach.

One more time . . . (trying tar this time)

Hmm, i do not see any "sqlite3 error:" lines in there?
Perhaps i failed to provide you with the right line, please try darktable -d sql &> /tmp/darktable-dbg-sql.txt

#14 Updated by David Wen Riccardi-Zhu about 2 years ago

Trying again. : \

Roman Lebedev wrote:

David Wen Riccardi-Zhu wrote:

Attached! File is ~175mib uncompressed.

File upload seems to be broken :(

Roman Lebedev wrote:

David Wen Riccardi-Zhu wrote:

David Wen Riccardi-Zhu wrote:

Looks like the file did not attach.

One more time . . . (trying tar this time)

Hmm, i do not see any "sqlite3 error:" lines in there?
Perhaps i failed to provide you with the right line, please try darktable -d sql &> /tmp/darktable-dbg-sql.txt

#15 Updated by Roman Lebedev about 2 years ago

  • Affected Version changed from 2.0.5 to git master branch
  • Subject changed from darktable crashing on import to SQLite issues. Was: darktable crashing on import
  • Status changed from Incomplete to Triaged
  • Priority changed from Low to Medium
  • System changed from Ubuntu to all

Also available in: Atom PDF