rclone

rclone copied to clipboard

Currently users don't know whether there is stuff in the VFS cache or not when quitting rclone.

Using --stats helps, but isn't a perfect solution.

From #1853

Although ideally a parameter that outputs if there's any transfer going on/files queued in cache that could be parsed in a script to check before unmounting or rebooting the server would be ideal!

Stuff still being left in the cache on quitting Rclone is just a matter of reclaiming disc space, the bigger issue IMO is making sure that changes are committed to the remote, so that Rclone isn't terminated before changes have had a chance to be written.

In addition to that it'd be nice if vfs and cache rc commands had a vfs/queue and cache/queue option to spit out what's left to upload and when.

@mstgrv It's not about read cache. It's about what's in the write cache (VFS queue). Not sure how gracefully the interruption and resuming of the upload from the queue (restarting the rclone process with data still in queue) is currently handled (which is of course important) but it's also important to be able to see if the queue has been successfully transferred to the remote for use cases where the delivery of files is time critical (eg I need to know if the queue has finished before I go into downtime for upgrading the server/upgrade linux version/even just a reboot to a new kernel that may cause additional downtime when the files need to be there now and not in possibly two hours). Currently --stats is all you can use for this purpose. An rclone parameter that would output the status of the queue in a clear way on stdout, so it could also be scripted, would be ideal.

Any update on this?

I'm using the following code as workaround:

rclone mount s3:my-bucket /mnt --rc --vfs-cache-mode minimal &

...

while rclone rc core/stats | jq -e '.transferring'; do
  sleep 1
done

wget -q -O- --method POST http://localhost:5572/core/quit

I'm using the following code as workaround:

rclone mount s3:my-bucket /mnt --rc --vfs-cache-mode minimal &

...

while rclone rc core/stats | jq -e '.transferring'; do
  sleep 1
done

wget -q -O- --method POST http://localhost:5572/core/quit

Thank you. I will use this workaround until flushing implemented.

I'm using the following code as workaround:

rclone mount s3:my-bucket /mnt --rc --vfs-cache-mode minimal &

...

while rclone rc core/stats | jq -e '.transferring'; do
  sleep 1
done

wget -q -O- --method POST http://localhost:5572/core/quit

This workaround fails when:

1. VFS write back is delayed
2. Error occurred in the process of write back, retries are queued

I made a similar workaround using vfs/stats.

while [ $( rclone rc vfs/stats | tr -d '\n' | sed -E 's/^\{(.*,)?\s*"diskCache"\s*:\s*\{(.*,)?\s*"(uploadsQueued|uploadsInProgress)"\s*:\s*([0-9]+)\s*(,.*)?,\s*"(uploadsQueued|uploadsInProgress)"\s*:\s*([0-9]+)\s*(,.*)?\}.*\}/\4 + \7/' | bc ) -gt 0 ] ; do
  sleep 1
done 

Sorry for the messy regex. The environment of rclone docker image doesn't have jq. This is the only way I found.

Thanks for sharing! :) I think the jq equivalent would be:

while [[ $(rclone rc vfs/stats | jq '.diskCache | [.uploadsInProgress, .uploadsQueued] | add') -gt 0 ]]; do
  sleep 1
done