sshfs icon indicating copy to clipboard operation
sshfs copied to clipboard

Support request size negotiation and increased throughput on high-latency connections

Open ccope opened this issue 2 years ago • 1 comments

I have found that a combination of increasing readahead and read limits to ~4MB has increased throughput from 1.1MB/s to 7.5MB/s on a connection with a 160ms round-trip time.

I need to submit another patch to the Linux kernel to allow configuring the readahead size. libfuse already seems to accept it as a parameter but the kernel side doesn't utilize it. Here's a patch that I used for testing which overrode the hard-coded limits (based on Ubuntu-hwe-5.4-5.4.0-79.88_18.04.1): https://gist.github.com/ccope/0ff1cac336fceb827696c40715a4a828

The OpenSSH SFTP server recently added request limit negotiation, but it also has a fairly restrictive hard-coded cap. I need to file a patch to make that limit more configurable as well.

I reworked how the sftp init process is invoked to ensure limits get updated before forking more threads. In the process I created some high-level helper functions for sending requests and receiving responses without threads, and used them for all of the sftp init requests. I'd be happy to move them to a separate PR but I just kind of got tired of reworking my git history.

ccope avatar Oct 01 '21 06:10 ccope

There seem to be some issues with this that causes processes to hang; it might be caused by deadlocks from the new sftp_request_process_sync and related functions. I am not sure exactly.

See https://bugzilla.suse.com/show_bug.cgi?id=1215574 and discussions here https://github.com/deadbeefsociety/sshfs/pull/1 (I assume it is WIP and hasn't been thoroughly tested)

h4sh5 avatar Sep 23 '23 12:09 h4sh5