Write performance are very slow

[mnencia]

Here is an example of the required to write 100Mb of random data

mnencia@spark:~/tmp/mchfuse$ time dd if=/dev/urandom of=test.bin count=100 bs=1M
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1175.19 s, 89.2 kB/s

real	19m36.021s
user	0m0.000s
sys	0m0.721s
mnencia@spark:~/tmp/mchfuse$ time dd of=/dev/null if=test.bin count=100 bs=1M
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 18.1986 s, 5.8 MB/s

real	0m18.201s
user	0m0.000s
sys	0m0.126s

I suppose that the issue resides in the size of chunks from the kernel interface. With the current implementation, every chunk requires a separate HTTP request. We can solve the problem by implementing some local buffering.

[jaybha]

<Disclaimer: this is first time I'm reading go, please ignore if irrelevant... I'd love to contribute once I familiarize myself with language>

Looks like the device module(?) calls auth token for every request, and that might be killing the performance. If there is a way to cache the JWT, to the point of its expiry, or get new one upon 401/403, local device access should certainly benefit

I was able to achieve much faster performance with this using postman API

[jlcordeiro]

I am not seeing this behaviour. Refresh is only called if it expires. Ran a few tests too and only one token was requested across all writes I did.

[jlcordeiro]

After further digging and testing I noticed that (at least on my system) Write was being called with 64kb chunks of data each time, even if the application was trying to write much bigger chunks. Each of them triggering a POST call (hence why it is so slow).

Bit of research and I've found out that both fuse and go-fuse allow you to set the size of the writes.

This PR uses the max value set in go-fuse. In my case this is now writting at 1Mb/s which is not great but is at least usable and won't drive me mad.

Note that the limit is still quite low (32 pages as you can see below). That said, at least this one-liner will make it as fast as fuse allows, and (assuming go-fuse gets updated) the speed will be increased whenever the limitation is lifted on fuse-kernel).

max_write=N
Set the maximum number of bytes in a single write operation. The default is 128kbytes. Note, that due to various limitations, the size of write requests can be much smaller (4kbytes). This limitation will be removed in the future.

On fully fixing it...

One option for anyone willing to go that far would be to recompile fuse and go-fuse and increasing the limit there. Not worth the risk though in my case.

Another is - as you said - to buffer it on the mchfuse side. My Go is not good enough to do it publicly though but I will give it a go in the upcoming weeks.

[jdewald]

@mnencia Would you be open to a PR which does the write buffer? I am currently testing a change on a fork which primarily makes two changes:

1. Introduces a write buffer (currently 2MB but will make it configurable)
2. Keeps resumable file "open" during the write, based on some observations where when MyCloud "commits" the file it seems to take a bit of time.

In my tests writing a 500MB file it was doing about 40MB/s (~300Mbps). #2 does mean that the size of the file is likely unobservable during writes, but I think this is ok as the device is meant more as file storage than sort of "online" usage. I can make that configurable as well.

Additionally my local changes also adds ability to do a "replace" rename. This is based on honoring the "FS_RENAME_NOREPLACE" flag (which fs-fuse ignores, but presumably passes through if set). I can file a separate issue for that though as it's really not related to write performance, it's just something I ran across as I use rsync for backups and it would break on files that already existed.