Allow utilizing multiple buckets as backend storage endpoints

[cayolblake]

What would you like to be added:

Allow utilizing multiple buckets as backend storage.

#=============

Why is this needed:

1. Workaround rate limits for many providers which is applied on the bucket level which will massively enhance I/O performance on the juicefs mount points.
2. Enhancing high-availability with possibility of replicating a single block to one or more bucket which ends up increasing uptime and reliability.
3. Allow using multiple providers which increases end-users service reliability and credibility.

#=============

[suzaku]

According to this S3 document,

your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket.

if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.

Reading and writing can be scaled by creating multiple prefixes in a bucket.

[cayolblake]

Well, assuming that's for AWS S3, it's not the case for the majority of other providers.

Also the other points in subject.

[davies]

Sounds good, we can have raid 1( replicated across multiple bucket/provider), and raid 0 (hash to multiple buckets).

The difficulty is to define a interface that's backward compatible and easy to understand, both on user interface and implementation (this is easy).

[cayolblake]

Yes totally agree.

RAID 0/1 like would give endless possibilities and mark juicefs for a new era of software defined file systems.

Also it would allow people to create outstanding ideas like layering a RAID 1 setup over a RAID 0 setup or vice versa to solve real-life problems in a smart easy way.

[davies]

RAID 0 is implemented in #349

[jkiebzak]

RAID 0 is implemented in #349

Great work with this. One question: In the case of running RAID 0 against local minio instances, is there a way to replace a minio server? Or would we be "stuck" with the DNS/IP addresses provided to the format command?

[davies]

We can use format command to update the DNS/IP, once we can make sure that the number and order of DNS/IP are the same as before.

[0akarma]

We can use format command to update the DNS/IP, once we can make sure that the number and order of DNS/IP are the same as before.

Does it means if it updates the original single DNS/IP to the load balance DNS/IP which points at the same bucket cluster, it will work?

[davies]

Does it means if it updates the original single DNS/IP to the load balance DNS/IP which points at the same bucket cluster, it will work?

yes

[eleaner]

Is it possible to utilise multiple endpoints in the famous RAID0? e.g. different providers

[akiirax]

@davies if there's a way to achieve that it would be great.