Discrepancy between speed test and real-world speeds when running locally

[knaccc]

I'm using gocryptfs v2.2.1 without_openssl, and the gocryptfs -speed results are:

gocryptfs v2.2.1 without_openssl; go-fuse v2.1.1-0.20210825171523-3ab5d95a30ae; 2021-10-24 go1.17.2 darwin/amd64
cpu: unknown; with AES acceleration
AES-GCM-256-OpenSSL       	    N/A
AES-GCM-256-Go            	3895.94 MB/s	(selected in auto mode)
AES-SIV-512-Go            	 325.05 MB/s
XChaCha20-Poly1305-OpenSSL	    N/A
XChaCha20-Poly1305-Go     	1856.94 MB/s	(selected in auto mode)

Diskmark speed tests report that my Macbook Pro SSD can perform sequential reads at 3.4GB/s and sequential writes at 2.1GB/s.

When I do cp random10G.bin random10G.bin.2, that completes at a speed of 1064 MB/s (this is just on the disk directly with gocryptfs not involved at all).

However, when I create local gocryptfs cipher and plain directories, I only get a write speed of 322 MB/s and a read speed of 312 MB/s to/from the gocryptfs plain directory.

I checked that AES-GCM-256 is being used:

gocryptfs -info cipher
Creator:           gocryptfs v2.2.1
FeatureFlags:      HKDF GCMIV128 DirIV EMENames LongNames Raw64
EncryptedKey:      64B
ScryptObject:      Salt=32B N=65536 R=8 P=1 KeyLen=32
contentEncryption: AES-GCM-256

Is it normal for gocryptfs to operate at a third of the speed of the underlying disk? Might there be any buffer size changes that could improve performance?

[rfjakob]

Can you check what you get on a ram disk?

It is expected that you get a fraction of your speed test result. I have 1000MB/s on the speed test and 500MB/s on a ramdisk mount.

[knaccc]

@rfjakob sure, I did the test twice as follows:

gocryptfs -speed
# gocryptfs v2.2.1 without_openssl; go-fuse v2.1.1-0.20210825171523-3ab5d95a30ae; 2021-10-24 go1.17.2 # darwin/amd64
# cpu: unknown; with AES acceleration
# AES-GCM-256-OpenSSL       	    N/A
# AES-GCM-256-Go            	3940.55 MB/s	(selected in auto mode)
# AES-SIV-512-Go            	 324.86 MB/s
# XChaCha20-Poly1305-OpenSSL	    N/A
# XChaCha20-Poly1305-Go     	1797.91 MB/s	(selected in auto mode)

diskutil erasevolume HFS+ 'ramdisk' `hdiutil attach -nobrowse -nomount ram://17000000`
cd /Volumes/ramdisk
time dd if=/dev/random of=4GB.bin bs=1 count=0 seek=4g
# completed in 3.445s = 1189MB/s  (3.268s on second run)

time cp 4GB.bin 4GB.bin.1
# completed in 12.592s = 325MB/s  (10.670s on second run)

rm 4GB.bin.1
mkdir cipher plain
gocryptfs -init cipher
gocryptfs cipher plain

time cp 4GB.bin plain/
# completed in 15.006s = 273 MB/s. (14.011s on second run)

I then repeated the test on my SSD (no ramdisk involved) as follows:

time dd if=/dev/random of=4GB.bin bs=1 count=0 seek=4g
# completed in 0.003s = 1365333 MB/s  (0.003s on second run)

time cp 4GB.bin 4GB.bin.1
# completed in 3.223s = 1271 MB/s  (4.054s on second run)

time cp 4GB.bin plain/
# completed in 12.030s = 340 MB/s. (14.675s on second run)

I'm thoroughly confused that when generating the random file from /dev/random, it takes 3.445s to write it to the ramdisk, but apparently 0.003s to write it to the SSD. I assume there must be some kind of write caching going on. It's also confusing that the ramdisk is slower than the SSD for gocryptfs.

Summary:

Regular (non-gocrypfts) 4GB file copy: 325 MB/s RAM Disk vs 1271 MB/s SSD 4GB file write into gocryptfs mount: 273 MB/s RAM Disk vs 340 MB/s SSD

[portasynthinca3]

I'm facing this issue too. The underlying filesystem is btrfs.

All tests performed with 1GiB files: Native SSD read: 6.1 GB/s Native SSD write: 2.3 GB/s gocryptfs read: 1.0 GB/s gocryptfs write: 211 MB/s

gocryptfs -speed: AES-GCM-256-Go: 4126.57 MB/s gocryptfs -info: contentEncryption: AES-GCM-256