gocryptfs
gocryptfs copied to clipboard
Improving transfer speed over network?
Hi
I tried Gocryptfs with sshfs and samba and in both cases the speed was plain horrible like couple kb/s, and I can send files to smb server upt couple mb/s so this is definetely not an issue with the network.
I created the cipher folder remotely. Then I mount the cipher folder (remote.cipher.cifs) over cifs locally. Finally I mount remote.cipher.cifs
to a local folder with gocryptfs remote.cipher.cifs /mnt/plain
It works but sooooo slowww. Is there a way to improvve the speed a bit?
There are 2 reasons why I am not using syncing
1- I do not want to double data locally, because I need top backup couple tb (like 5 tb) to the other side , so that means I need to keep all that in separate syncable disk space
2- I never want to mount the folder plain on the remote since 1
being necessity that is my only option, however I do not want to mount it on the remote pc. So I seem to have no other option given that the speed is horrible with using rsync
to cp backups to cifs mounted plain folder.
I am ok with if I can get around 400 kb to 1mb
Hi, thanks for the report! I use it regularily over nfs, speeds are ok, but i will test with cifs.
What are your ping times to the cifs server?
Hi
My ping time is about 160 ms, not too fast.
I was trying to back up a folder that was about 10mb, around 1500 files. Rsync takes so long (15+? min) "to start" that I could not even wait that long. Once I waited long enough for that folder to see what would happen, the file transfer was couple kb a sec. So I cancelled it since that would never finish. And "Iotop" was reporting about %25 of IO by Gocryptfs on the local Debian pc. Not sure why it was reporting that much of IO given that there was not even any data going back and forth by Rsync, but it is possible I guess.
Btw what settings do you use with NFS on both ends? I can also try that maybe
Here is a test over NFS :(
sent 3.56M bytes received 31.98K bytes 916.51 bytes/sec
total size is 3.42M speedup is 0.95
And here is Rsync copying to the same remote over SSH with speed limited to "500kb" via Rsync's --bwlimit
As you se e it can output 500kb steaily no problem
FOLDER/data/1/1315
434.68M 100% 499.76kB/s 0:14:09 (xfr#783, to-chk=561/1349)
FOLDER/data/1/1323
107.28M 24% 500.24kB/s 0:10:51
I am not sure what my speed is, but I use gocryptfs
for a home directory mounted on the local network via kerberized NFSv4 every day (sec=krb5i
). This is on Debian stable.
I tested with sshfs, and I added 100ms of extra latency like this:
tc qdisc add dev eth0 root netem delay 100ms
Now the ping time to my server is about 125ms.
I don't see a big difference between plain sshfs and gocryptfs on top of sshfs:
$ time cp /tmp/10mb sshfs.mnt/
real 0m27.733s
$ time cp /tmp/10mb sshfs.gocryptfs.mnt/
real 0m33.515s
Ok, there is a big difference between sshfs and gocryptfs on top of sshfs when running rsync with many small files. I am testing with a new empty git repo created like this:
git init newgit
$ time rsync -r newgit sshfs.mnt/
real 0m4.073s
$ time rsync -r newgit sshfs.gocryptfs.mnt/
real 0m17.132s
But you can speed it up a lot by using the --inplace
option for ssh:
$ time rsync -r --inplace newgit sshfs.gocryptfs.mnt/
real 0m1.567s
PS: Make sure to compile the latest version from git. Changes from https://github.com/rfjakob/gocryptfs/issues/410 should give you a big speedup as well.
I did not get much of a relief from inplace option, maybe it did hard to tell
time rsync -rvPh --delete --inplace folder /mnt/plain.cifs
sent 3.55M bytes received 57.89K bytes 1.41K bytes/sec
total size is 3.42M speedup is 0.95
real 42m44.294s
user 0m0.255s
sys 0m0.805s
I will compile and try the new version.
Btw will I need to upgrade file system regularly with version upgrades?
Not much progress, this time I tried it over with Nfs with 1.8
time rsync -rvh --delete --inplace --no-perms --no-owner --no-group FOLDER /mnt/plain.nfs
sent 3.56M bytes received 31.92K bytes 1.18K bytes/sec
total size is 3.42M speedup is 0.95
real 50m46.473s
user 0m0.221s
sys 0m1.022s
find FOLDER -type f |wc -l
1606
du -sh FOLDER
9.9M FOLDER
Btw will I need to upgrade file system regularly with version upgrades?
No, the file format on disk has not changed for a long time now. No action needed for your file systems.
Still not very good.
$ gocryptfs -version
gocryptfs v2.0-beta3; go-fuse v2.1.1-0.20210423170155-a90e1f463c3f; 2021-05-03 go1.15.2 linux/amd64
gocryptfs/tests $ ./sshfs-benchmark.bash nuetzlich.net
working directory: /tmp/sshfs-benchmark.bash.xsm
sshfs mounted: nuetzlich.net:/tmp -> sshfs.mnt
gocryptfs mounted: sshfs.mnt/sshfs-benchmark.bash.QAW/gocryptfs.crypt -> gocryptfs.mnt
sshfs-benchmark.bash: sshfs gocryptfs-on-sshfs
git init 8.48 17.60
rsync 15.04 26.66
rm -R 4.83 20.78
mkdir 2.50 7.56
rmdir 0.82 7.39
touch 4.22 5.32
rm 0.81 0.81
Is there any way to disable the safe path walking / fix introduced in v1.7 when not using plaintextnames
option? I wanted to avoid going back to v1.6.1 from v2.2.1, seeing there are lots of other fixes implemented since and I compiling v1.6.1 seems non-trivial (new instructions don't work old rep, old instructions don't work with recent go version)
In my case (NFS mounted NAS) I loose about 20x speed with gocryptfs vs. plain text, pv /dev/urandom > test.img
gives me 2-5 MiB/s while I get ~90 MiB/s (gigabit ethernet) without encryption.
If you are one Intel/AMD, there's a binary here: https://github.com/rfjakob/gocryptfs/releases/tag/v1.6.1
HOWEVER, the safe path walking should not impact streaming writes like you tested above. Something else seems to be going on here.
On my machine and a Synology NAS mounted over nfs:
gocryptfs on nfs:
$ dd if=/dev/urandom of=blob bs=128k count=100 conv=fdatasync
100+0 records in
100+0 records out
13107200 bytes (13 MB, 12 MiB) copied, 0,43358 s, 30,2 MB/s
plain nfs:
$ dd if=/dev/urandom of=blob bs=128k count=100 conv=fdatasync
100+0 records in
100+0 records out
13107200 bytes (13 MB, 12 MiB) copied, 0,279807 s, 46,8 MB/s
$ gocryptfs --version
gocryptfs v2.2.1-31
I'm using dd conv=fdatasync
because the results I got with pv
were not plausible (270MB/s over gigabit, yeah, right. Plain nfs writes go to a huge write buffer first)
Thank you for checking! I just tried to check it (with dd
) in a few different configs, on an external SSD and the NAS with both versions as well as plain (no gocryptfs).
TL,DR: Still about 60% slower with 2.2.1 compared t 1.6.1 but not as extreme as thought. Edit: Looks like 1.7.1 behaves similarly to 1.6.1 for me, so it should actually be a different change? I'll try release-bisect :)
NFS(v3) NAS (Old desktop, Gigabit ethernet, 3x Crucial BX500 in ZFS raidz)
plain:
dd if=/tmp/test_urandom.dat of=plain/test.img bs=128k conv=fdatasync status=progress
8000+0 records in [cutting out these two lines in the next outputs]
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 11.6189 s, 90.2 MB/s [90.4, 112, 111, 113]
Now with gocryptfs:
mkdir gocrypt_221; mkdir gocrypt_221_mnt; /usr/bin/gocryptfs -init gocrypt_221/; /usr/bin/gocryptfs gocrypt_221 gocrypt_221_mnt/
mkdir gocrypt_161; mkdir gocrypt_161_mnt; ~/dls/gocryptfs -init gocrypt_161/; ~/dls/gocryptfs gocrypt_161 gocrypt_161_mnt/
gocryptfs 2.2.1:
dd if=/tmp/test_urandom.dat of=gocrypt_221_mnt/test.img bs=128k conv=fdatasync
1048576000 bytes (1.0 GB, 1000 MiB) copied, 23.957 s, 43.8 MB/s [44.8, 22.7, 43.2, 44.6]
gocryptfs 1.6.1:
dd if=/tmp/test_urandom.dat of=gocrypt_161_mnt/test.img bs=128k conv=fdatasync
1048576000 bytes (1.0 GB, 1000 MiB) copied, 10.1908 s, 103 MB/s [102, 104, 102, 91.6]
I repeated the test a few times (numbers in backets). The CPU usage during the copy on my computer was around 10%, and on the NAS around 5%. Those numbers looked higher than than the pv
numbers recently, so I did that again and it seems the values were higher than before but still different -- let's focus on dd
though to avoid buffers & caches.
pv /dev/urandom > gocrypt_161_mnt/test.img [78.5MiB/s]
pv /dev/urandom > gocrypt_221_mnt/test.img [28.6MiB/s]
External SSD (Samsung EVO, USB adaptor, ext4): Unmount & redo everything on a folder on the external SSD. Plain:
dd if=/tmp/test_urandom.dat of=plain/test.img bs=128k conv=fdatasync status=progress
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.70894 s, 387 MB/s
gocryptfs 2.2.1:
dd if=/tmp/test_urandom.dat of=gocrypt_221_mnt/test.img bs=128k conv=fdatasync
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.58102 s, 293 MB/s
gocryptfs 1.6.1:
dd if=/tmp/test_urandom.dat of=gocrypt_161_mnt/test.img bs=128k conv=fdatasync
8000+0 records in
8000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.52897 s, 297 MB/s
Edit2: Just did an rsync of my home-dir to the NAS (gocryptfs 2.2.1) and got 22.4 MB/s over 730 GB; can test with 1.6.1 if helpful but I think we can first debug the sequential case.
Okay I just did a quick check through versions, and it looks like the slow-down came between 1.8 and 2.0-beta1:
gocryptfs v1.8.0 without_openssl; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-05-09 go1.14.2 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 10.4414 s, 100 MB/s
gocryptfs v2.0-beta1.HEAD without_openssl; go-fuse v2.0.4-0.20200908172753-0b6cbc515082 => github.com/rfjakob/go-fuse/v2 v2.0.4-0.20201015204057-88b12c99f8af; 2020-10-19 go1.15.3 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 24.7494 s, 42.4 MB/s
gocryptfs v2.2.1 without_openssl; go-fuse v2.1.1-0.20210825171523-3ab5d95a30ae; 2021-10-20 go1.17.2 linux/amd64
1048576000 bytes (1.0 GB, 1000 MiB) copied, 23.6018 s, 44.4 MB/s
Test-script:
mkdir crypt; mkdir mnt;
~/dls/gocryptfs -init crypt
~/dls/gocryptfs crypt mnt
~/dls/gocryptfs --version
dd if=/tmp/test_urandom.dat of=mnt/test.img bs=128k conv=fdatasync
fusermount -u mnt
rm -r mnt crypt
Thanks for the benchmarks! The low cpu usage suggests that it is latency-bound. Maybe my NAS is too slow to see the effect as strongly as you do. I'll try to get something faster (nfs mount of 127.0.0.1 maybe?)
This should be fixed now
Right, via https://github.com/rfjakob/gocryptfs#v231-2023-03-04 / https://github.com/rfjakob/gocryptfs/commit/8f3ec5dcaa6eb18d11746675190a7aaceb422764