S3 Remote: Use Multiple Connections

[michaelkoetter]

Is your feature request related to a problem? Please describe. Some S3 compatible services like Backblaze are rather slow on a single connection. To speed them up, they require using multiple concurrent connections / transfers (like rclone --transfers ... flag).

Describe the solution you'd like The S3 remote should support using multiple connections. The number of connections should be configurable.

Describe alternatives you've considered Live with painfully slow S3 backups :)

Additional info Example Backup Job from Hetzner (Falkenstein, Germany FSN1) to Backblaze EU (Amsterdam) shows approximately 50 Megabit/s upload bandwidth, whereas the link should be capable of 1 Gigabit/s and I have seen rclone transfers to Backblaze achieving almost 100% link saturation.

[julien-f]

It should already be using multiple connections, I'll discuss this with @fbeauchamp next week and will keep you posted.

[fbeauchamp]

the link from XO to the S3 sotrage should use 16 concurrent connections. What is sthe real export speed of your XCP/Xen host ? for example with a local backup Also, the block are compressed with brotli before being sent. Is the CPU usage of the XO VM too high during backup ?