Project

General

Profile

Bug #11689

Darktable truncate large files to 2GB when importing

Added by Diogo Sousa 2 months ago. Updated about 1 month ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
General
Target version:
Start date:
08/12/2017
Due date:
% Done:

100%

Affected Version:
2.2.5
System:
other GNU/Linux
bitness:
64-bit
hardware architecture:
amd64/x86

Description

When importing large files (in this case videos) darktable truncates them to 2GB. In my case this happened when "Importing from camera".

Associated revisions

Revision f3a7cc6e
Added by Diogo Sousa about 1 month ago

Fix partial copy of large files on "import from camera".

This happened because `write()` makes no guarantees of writing everything that
was requested. Apparently, in Linux, it doesn't write more than 2GB at a time,
but writing a file of any size is only correct if we keep calling `write()`
until everything is written.

Closes bug #11689.

History

#1 Updated by Roman Lebedev 2 months ago

Diogo Sousa wrote:

When importing large files (in this case videos) darktable truncates them to 2GB. In my case this happened when "Importing from camera".

So.. Uhm.. Are those raw (MLV) videos?

#2 Updated by Diogo Sousa 2 months ago

Roman Lebedev wrote:

Diogo Sousa wrote:

When importing large files (in this case videos) darktable truncates them to 2GB. In my case this happened when "Importing from camera".

So.. Uhm.. Are those raw (MLV) videos?

No, this is just a MP4 file. I think darktable treated this as "just a file", i.e. if it was a big "wikipedia-dump.xml" it would also truncate it, I think (I can test this if it helps.).

#3 Updated by Roman Lebedev 2 months ago

Diogo Sousa wrote:

Roman Lebedev wrote:

Diogo Sousa wrote:

When importing large files (in this case videos) darktable truncates them to 2GB. In my case this happened when "Importing from camera".

So.. Uhm.. Are those raw (MLV) videos?

No, this is just a MP4 file. I think darktable treated this as "just a file", i.e. if it was a big "wikipedia-dump.xml" it would also truncate it, I think (I can test this if it helps.).

So why are you using darktable to download it then? (And, why is everyone using that 'import from camera' functionality?)
I would guess it's this code:
https://github.com/darktable-org/darktable/blob/42b882600c3f96714b0ad6c19524c98c21f74df7/src/common/camera_control.c#L915-L950
unsigned long int size; <- so does gphoto itself even properly handle such big files?

#4 Updated by Diogo Sousa 2 months ago

Roman Lebedev wrote:

So why are you using darktable to download it then? (And, why is everyone using that 'import from camera' functionality?)

If darktable supports importing non-photo stuff it should do so correctly. It should either not import unknown file formats or copy them correctly.

https://github.com/darktable-org/darktable/blob/42b882600c3f96714b0ad6c19524c98c21f74df7/src/common/camera_control.c#L915-L950
unsigned long int size; <- so does gphoto itself even properly handle such big files?

That unsigned long int usually have 64 bits if compiled for a 64 bits computer (which is my case), so it might not be that. Anyway, the type to hold the file size should probably be off_t.

#6 Updated by Tobias Ellinghaus about 1 month ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

#7 Updated by Roman Lebedev about 1 month ago

  • Target version set to 2.4.0

Also available in: Atom PDF