Performance configuration for uploading ~30 TB

[acardona]

Now that it's running nicely, I am estimating it will take about 60 days to copy 30 TB worth of data (several hundred thousand files) using a single-threaded rsync.

What would be the preferred approach? Any configuration options to optimize performance? Is it effective to use multiple rsync processes, given that google-drive-ocamlfuse is multi-threaded and our bandwidth is nowhere near saturated?

Thank you!

[astrada]

I've never tried to upload such a big archive. If you are feeling lucky write_buffers=true enables memory caching for block writing, and that should speed things up somewhat. But then you have to verify that the upload was successful checking file hashes or checksum because that feature is still experimental. But probably your best option is to use rclone or drive to do the first upload, because they are optimized for that scenario (file sync).

[santosadrian]

I put my issue here because it's related. I'm getting very slow speed uploading a few text files https://i.imgur.com/CfweBpK.png How can I improve it?

[TheRealFanjin]

Yeah, I'm having the same problem. I'm getting about 250 bytes per second, and it takes forever to upload a 95 MB folder.