sshfs
sshfs copied to clipboard
sshfs is much slower than rclone mount over high-latency connections due to lack of SFTP pipelining
I know that sshfs is not currently actively developed, but writing this down here because so many people have had problems with this topic and it is not properly documented here in the upstream project.
Observation
cat
ing a file over sshfs
can be 100x slower over rclone mount
with rclone's SFTP backend over high-latency connectinons (servers far away).
For example, on a 200 ms connection, pv < ssh-mount/myfile > /dev/null
only prouces 1 MB/s when the connection shows 100 MB/s over TCP (measured with iperf3
).
Reason
- The SSH protocol specifies a minimum packet size that all implementations must support.
- OpenSSH implements a maximum packet size of 256 KiB.
- See link above and
#define PACKET_MAX_SIZE (256 * 1024)
- This is quite annoying because it effectively disables TCP Window Scaling, which was designed to provide high throughput over high-latency TCP links, by allowing more data to be in flight:
- While the TCP would allow more data to be in flight, the OpenSSH implementation will just not send more data.
- See link above and
- The SFTP protocol has a mitigation for this:
In general, this protocol follows a simple request-response model. Each request and response contains a sequence number and multiple requests may be pending simultaneously. spec source
- This concept is often called pipelining.
-
rclone
implements pipelining with its SFTPconcurrency
setting, defaulting to64
.- Implemented by the Go library
pkg/sftp
in this code:// Split the read into multiple maxPacket-sized concurrent reads bounded by maxConcurrentRequests. // This allows writes with a suitably large buffer to transfer data at a much faster rate // by overlapping round trip times.
- Implemented by the Go library
sshfs
does not seem to implement request pipelining, thus is is much slower.