Improving transfer speed over network?

[gerroon]

Hi

I tried Gocryptfs with sshfs and samba and in both cases the speed was plain horrible like couple kb/s, and I can send files to smb server upt couple mb/s so this is definetely not an issue with the network.

I created the cipher folder remotely. Then I mount the cipher folder (remote.cipher.cifs) over cifs locally. Finally I mount remote.cipher.cifs to a local folder with gocryptfs remote.cipher.cifs /mnt/plain

It works but sooooo slowww. Is there a way to improvve the speed a bit?

There are 2 reasons why I am not using syncing

1- I do not want to double data locally, because I need top backup couple tb (like 5 tb) to the other side , so that means I need to keep all that in separate syncable disk space

2- I never want to mount the folder plain on the remote since 1 being necessity that is my only option, however I do not want to mount it on the remote pc. So I seem to have no other option given that the speed is horrible with using rsync to cp backups to cifs mounted plain folder.

I am ok with if I can get around 400 kb to 1mb

[rfjakob]

Hi, thanks for the report! I use it regularily over nfs, speeds are ok, but i will test with cifs.

What are your ping times to the cifs server?

[gerroon]

Hi

My ping time is about 160 ms, not too fast.

I was trying to back up a folder that was about 10mb, around 1500 files. Rsync takes so long (15+? min) "to start" that I could not even wait that long. Once I waited long enough for that folder to see what would happen, the file transfer was couple kb a sec. So I cancelled it since that would never finish. And "Iotop" was reporting about %25 of IO by Gocryptfs on the local Debian pc. Not sure why it was reporting that much of IO given that there was not even any data going back and forth by Rsync, but it is possible I guess.

Btw what settings do you use with NFS on both ends? I can also try that maybe

[gerroon]

Here is a test over NFS :(

sent 3.56M bytes  received 31.98K bytes  916.51 bytes/sec
total size is 3.42M  speedup is 0.95

And here is Rsync copying to the same remote over SSH with speed limited to "500kb" via Rsync's --bwlimit As you se e it can output 500kb steaily no problem

FOLDER/data/1/1315
        434.68M 100%  499.76kB/s    0:14:09 (xfr#783, to-chk=561/1349)
FOLDER/data/1/1323
        107.28M  24%  500.24kB/s    0:10:51  

[lechner]

I am not sure what my speed is, but I use gocryptfs for a home directory mounted on the local network via kerberized NFSv4 every day (sec=krb5i). This is on Debian stable.

[rfjakob]

I tested with sshfs, and I added 100ms of extra latency like this:

tc qdisc add dev eth0 root netem delay 100ms

Now the ping time to my server is about 125ms.

I don't see a big difference between plain sshfs and gocryptfs on top of sshfs:

$ time cp /tmp/10mb sshfs.mnt/
real	0m27.733s

$ time cp /tmp/10mb sshfs.gocryptfs.mnt/
real	0m33.515s

[rfjakob]

Ok, there is a big difference between sshfs and gocryptfs on top of sshfs when running rsync with many small files. I am testing with a new empty git repo created like this:

git init newgit

$ time rsync -r newgit sshfs.mnt/
real	0m4.073s

$ time rsync -r newgit sshfs.gocryptfs.mnt/
real	0m17.132s

But you can speed it up a lot by using the --inplace option for ssh:

$ time rsync -r --inplace newgit sshfs.gocryptfs.mnt/
real	0m1.567s

[rfjakob]

PS: Make sure to compile the latest version from git. Changes from https://github.com/rfjakob/gocryptfs/issues/410 should give you a big speedup as well.

[gerroon]

I did not get much of a relief from inplace option, maybe it did hard to tell

time rsync -rvPh --delete --inplace folder /mnt/plain.cifs

sent 3.55M bytes  received 57.89K bytes  1.41K bytes/sec
total size is 3.42M  speedup is 0.95
real    42m44.294s
user    0m0.255s
sys     0m0.805s

I will compile and try the new version.

Btw will I need to upgrade file system regularly with version upgrades?

[gerroon]

Not much progress, this time I tried it over with Nfs with 1.8

time rsync -rvh --delete --inplace --no-perms --no-owner --no-group FOLDER /mnt/plain.nfs

sent 3.56M bytes  received 31.92K bytes  1.18K bytes/sec
total size is 3.42M  speedup is 0.95

real    50m46.473s
user    0m0.221s
sys     0m1.022s

find FOLDER -type f |wc -l
1606

du -sh FOLDER
9.9M    FOLDER

[rfjakob]

Btw will I need to upgrade file system regularly with version upgrades?

No, the file format on disk has not changed for a long time now. No action needed for your file systems.

[rfjakob]

Still not very good.

$ gocryptfs -version
gocryptfs v2.0-beta3; go-fuse v2.1.1-0.20210423170155-a90e1f463c3f; 2021-05-03 go1.15.2 linux/amd64

gocryptfs/tests $ ./sshfs-benchmark.bash nuetzlich.net
working directory: /tmp/sshfs-benchmark.bash.xsm
sshfs mounted: nuetzlich.net:/tmp -> sshfs.mnt
gocryptfs mounted: sshfs.mnt/sshfs-benchmark.bash.QAW/gocryptfs.crypt -> gocryptfs.mnt

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  8.48               17.60
rsync                    15.04               26.66
rm -R                     4.83               20.78
mkdir                     2.50                7.56
rmdir                     0.82                7.39
touch                     4.22                5.32
rm                        0.81                0.81

[Stefan-Heimersheim]

Is there any way to disable the safe path walking / fix introduced in v1.7 when not using plaintextnames option? I wanted to avoid going back to v1.6.1 from v2.2.1, seeing there are lots of other fixes implemented since and I compiling v1.6.1 seems non-trivial (new instructions don't work old rep, old instructions don't work with recent go version)

In my case (NFS mounted NAS) I loose about 20x speed with gocryptfs vs. plain text, pv /dev/urandom > test.img gives me 2-5 MiB/s while I get ~90 MiB/s (gigabit ethernet) without encryption.

[rfjakob]

If you are one Intel/AMD, there's a binary here: https://github.com/rfjakob/gocryptfs/releases/tag/v1.6.1

[rfjakob]

HOWEVER, the safe path walking should not impact streaming writes like you tested above. Something else seems to be going on here.

On my machine and a Synology NAS mounted over nfs:

gocryptfs on nfs:
$ dd if=/dev/urandom of=blob bs=128k count=100 conv=fdatasync
100+0 records in
100+0 records out
13107200 bytes (13 MB, 12 MiB) copied, 0,43358 s, 30,2 MB/s

plain nfs:
$ dd if=/dev/urandom of=blob bs=128k count=100 conv=fdatasync
100+0 records in
100+0 records out
13107200 bytes (13 MB, 12 MiB) copied, 0,279807 s, 46,8 MB/s

$ gocryptfs --version
gocryptfs v2.2.1-31

I'm using dd conv=fdatasync because the results I got with pv were not plausible (270MB/s over gigabit, yeah, right. Plain nfs writes go to a huge write buffer first)

[Stefan-Heimersheim]

Thank you for checking! I just tried to check it (with dd) in a few different configs, on an external SSD and the NAS with both versions as well as plain (no gocryptfs).

TL,DR: Still about 60% slower with 2.2.1 compared t 1.6.1 but not as extreme as thought. Edit: Looks like 1.7.1 behaves similarly to 1.6.1 for me, so it should actually be a different change? I'll try release-bisect :)

NFS(v3) NAS (Old desktop, Gigabit ethernet, 3x Crucial BX500 in ZFS raidz)

plain:

dd if=/tmp/test_urandom.dat of=plain/test.img bs=128k conv=fdatasync status=progress
8000+0 records in [cutting out these two lines in the next outputs]
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 11.6189 s, 90.2 MB/s [90.4, 112, 111, 113]

Now with gocryptfs:

mkdir gocrypt_221; mkdir gocrypt_221_mnt; /usr/bin/gocryptfs -init gocrypt_221/;  /usr/bin/gocryptfs gocrypt_221 gocrypt_221_mnt/
mkdir gocrypt_161; mkdir gocrypt_161_mnt;  ~/dls/gocryptfs -init gocrypt_161/; ~/dls/gocryptfs gocrypt_161 gocrypt_161_mnt/

gocryptfs 2.2.1:

dd if=/tmp/test_urandom.dat of=gocrypt_221_mnt/test.img bs=128k conv=fdatasync
1048576000 bytes (1.0 GB, 1000 MiB) copied, 23.957 s, 43.8 MB/s [44.8, 22.7, 43.2, 44.6]

gocryptfs 1.6.1:

dd if=/tmp/test_urandom.dat of=gocrypt_161_mnt/test.img bs=128k conv=fdatasync
1048576000 bytes (1.0 GB, 1000 MiB) copied, 10.1908 s, 103 MB/s [102, 104, 102, 91.6]

I repeated the test a few times (numbers in backets). The CPU usage during the copy on my computer was around 10%, and on the NAS around 5%. Those numbers looked higher than than the pv numbers recently, so I did that again and it seems the values were higher than before but still different -- let's focus on dd though to avoid buffers & caches.

pv /dev/urandom > gocrypt_161_mnt/test.img [78.5MiB/s]
pv /dev/urandom > gocrypt_221_mnt/test.img [28.6MiB/s]

External SSD (Samsung EVO, USB adaptor, ext4): Unmount & redo everything on a folder on the external SSD. Plain:

dd if=/tmp/test_urandom.dat of=plain/test.img bs=128k conv=fdatasync status=progress
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.70894 s, 387 MB/s

gocryptfs 2.2.1:

dd if=/tmp/test_urandom.dat of=gocrypt_221_mnt/test.img bs=128k conv=fdatasync
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.58102 s, 293 MB/s

gocryptfs 1.6.1:

dd if=/tmp/test_urandom.dat of=gocrypt_161_mnt/test.img bs=128k conv=fdatasync
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.52897 s, 297 MB/s

Edit2: Just did an rsync of my home-dir to the NAS (gocryptfs 2.2.1) and got 22.4 MB/s over 730 GB; can test with 1.6.1 if helpful but I think we can first debug the sequential case.

[Stefan-Heimersheim]

Okay I just did a quick check through versions, and it looks like the slow-down came between 1.8 and 2.0-beta1:

gocryptfs v1.8.0 without_openssl; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-05-09 go1.14.2 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 10.4414 s, 100 MB/s

gocryptfs v2.0-beta1.HEAD without_openssl; go-fuse v2.0.4-0.20200908172753-0b6cbc515082 => github.com/rfjakob/go-fuse/v2 v2.0.4-0.20201015204057-88b12c99f8af; 2020-10-19 go1.15.3 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 24.7494 s, 42.4 MB/s

gocryptfs v2.2.1 without_openssl; go-fuse v2.1.1-0.20210825171523-3ab5d95a30ae; 2021-10-20 go1.17.2 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 23.6018 s, 44.4 MB/s

Test-script:

mkdir crypt; mkdir mnt;
~/dls/gocryptfs -init crypt
~/dls/gocryptfs crypt mnt
~/dls/gocryptfs --version
dd if=/tmp/test_urandom.dat of=mnt/test.img bs=128k conv=fdatasync
fusermount -u mnt
rm -r mnt crypt

[rfjakob]

Thanks for the benchmarks! The low cpu usage suggests that it is latency-bound. Maybe my NAS is too slow to see the effect as strongly as you do. I'll try to get something faster (nfs mount of 127.0.0.1 maybe?)

[jumoog]

This should be fixed now

[rfjakob]

Right, via https://github.com/rfjakob/gocryptfs#v231-2023-03-04 / https://github.com/rfjakob/gocryptfs/commit/8f3ec5dcaa6eb18d11746675190a7aaceb422764