Project

General

Profile

Bug #12102

Very slow performance over SMB

Added by Bojidar Bojkov 3 months ago. Updated 15 days ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Lighttable
Target version:
-
Start date:
03/29/2018
Due date:
% Done:

0%

Affected Version:
2.4.0
System:
Ubuntu
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Importing images over SMB connection is very slow.

My os is Ubuntu 16.04, Dell laptop (from 2011). 8GB RAM

My mounting options are

//172.16.30.3/t_drive /media/bojo/netgear1 cifs credentials=/home/bojo/bin/.smbcredentials,iocharset=utf8,sec=ntlm,uid=1001,gid=1001
0 0

I am attaching an USB3 drive plugged to a Netgear r7000 router. The drive is Seagate 3TB formatted with EXT4.

I observed that in order to import 4400 images it took 8+ hours. If I am to attach the drive directly to the laptop - same images would import in about 30 min.

I noticed that each image imports in 5-6 sec. Some times however - it stalls and it takes about 70 sec to import an image. I noticed that even when it was working at its fastest speed - only 2.5 to 5MB/s would be consumed from the bandwidth on read. The CPU is also not over utilized. If I am to copy the same images with corresponding xml files (using Nautilus) I can reach speeds of up to 72 MB/s. 5 GB if images would be copied in 1 min 15 s and 20 GB would be copied in 4 min 30 s.

As I do not have access to NFS - I cannot check if the behaviour is strictly SMB related or it affects any network stored images.

DT is 2.4.2 installed from
http://ppa.launchpad.net/pmjdebruijn/darktable-release/ubuntu

History

#1 Updated by Andreas Schneider 3 months ago

Does the Netgear router support SMB2 or SMB3 and if yes, is it enabled?

You can find out using:

smbclient -mSMB3 -d10 //172.16.30.3/t_drive

and then search the output for:

negotiated dialect

#2 Updated by Andreas Schneider 3 months ago

Also as you mount the SMB share using mount.cifs this is not an issue of darktable but probably of the Netgear SMB server. If they use Samba then it might be that they either do not support SMB2+ or that they have their own implementation which doesn't support leases. I don't think that darktable can help you here.

You should probably use the following options:

sec=ntlmssp
vers=3.1.1

if possible for mounting.

#3 Updated by Bojidar Bojkov 3 months ago

Just wanted to say Thank You...
My current setting is
//172.16.30.3/t_drive /media/bojo/netgear1 cifs credentials=/home/bojo/bin/.smbcredentials,iocharset=utf8,sec=ntlmssp,vers=3.0,uid=1001,gid=1001 0 0
vers=3.1.1 was not accepted (I got an error) but 3.0 was
I am still surprised that it works as everything that I am finding about r7000 says that it uses SMB1 and there is no specific setting on the router about SMB version.
I did a test with 600 images and they imported in 10 min. with an image showing up about one per second. That means - it should import 4400 images in about 1.5 hours (as a comparison of 8+h).
I am guessing I am reaching the limit of what my hardware can provide.
Thank you for your help!

#4 Updated by Jan Müller 22 days ago

Try NFS of possible, it could be faster.

smb speed is very CPU depended and its always too slow

#5 Updated by Andreas Schneider 22 days ago

Jan, why should smb be more CPU bound than NFS? You base that on what?

#6 Updated by Bojidar Bojkov 15 days ago

I am a bit perplexed but I did modify the setting recently.

From:
//172.16.30.3/t_drive /media/bojo/netgear1 cifs credentials=/home/bojo/bin/.smbcredentials,iocharset=utf8,sec=ntlmssp,vers=3.0,uid=1001,gid=1001 0 0

To:
//172.16.30.3/t_drive /media/bojo/netgear1 cifs credentials=/home/bojo/bin/.smbcredentials,iocharset=utf8,sec=ntlmssp,vers=1.0,uid=1001,gid=1001 0 0

Reasons:
  • I noticed that while I was getting very good "read" speed from the router (70+ MB/s in the file manager) I was getting very poor "write" speeds - about 5MB/s
  • In DT I was not assessing the speed well enough because DT would only modify the .XMP (and it is very small)
  • Since the R7000 does not have specific setting for SMB level - and it is already few years old - most likely it is SMB1 (I can only guess).

Since the change - the Read speed dropped a bit but the Write speed increased significantly - up to 25 MB/s (recalling loosely).
Today I repeated the same test of 600 images an it completed in 6m 43s. (it even improved from before).

My current thoughts are:
  • Maybe my initial declaration for SMB mount was not very good (originally I was not specifying the exact SMB version)
  • It is possible that there were other changes that happen on the Linux system that affected the behaviour. Most likely I would never know.
  • I am posting here if somebody stumbles on the same issue - my notes may be of some help.
  • I did not observe some CPU overload - All tests were done with quite loaded system. I had 7.5 GB RAM consumed out of 8GB total and 450 MB SWAP on SSD. Numerous programs running. CPU was 100-200% - the scale is registering up to 800% (8 cores)
  • I cannot compare to NFS because I don't have such a resource currently. What I have heard in the past is that NFS is very fast (and I should admit I had very good experience in the past when it comes to speed). In the current config however - the router acting as a file server I cannot implement it (because there is no NFS protocol on it).

Also available in: Atom PDF