s3proxy icon indicating copy to clipboard operation
s3proxy copied to clipboard

Middleware - sharding to multiple backends

Open Jayd603 opened this issue 3 years ago • 2 comments

Is this currently possible? I see I can shard across virtual buckets which is nice but it's not clear on if it would allow sharding across multiple backends (in this case it would be identical backends using the file system). That feature would be great.

Although now that I think about it - technically it would be the same backend type (file system) but if i could at least choose different backend paths for the sharding that could work.

Jayd603 avatar Oct 31 '22 14:10 Jayd603

This is a reasonable feature request and something the startup Bounce Storage implemented. It is easy to make S3Proxy write to multiple backends, at least when all the putBlob requests succeed. However it is more difficult to create policies for reading from a backend. Does it only read from one of them? Does it try to read from one and fail over to the second? What happens when writes fail to only one of the backends?

gaul avatar Jan 30 '23 05:01 gaul