Explain how to configure ClickHouse on S3, with some potential caveats.

[alexey-milovidov]

Set up TTL for incomplete multipart uploads. Do not set up a lifecycle policy for object deletion. Do not set up object versioning. Do not set up bucket replication. Do not use FUSE over S3. Do not point multiple shards to the same prefix in S3. Do not delete objects on S3 manually. Use separate buckets for ClickHouse and for other workloads. How to properly configure the S3 bucket policy to allow the needed operations. How to properly configure the instance IAM profiles. How KMS affects performance. Multi-region vs. single region buckets in GCS. Choosing the proper region for a bucket. Caveats of using R2 and B2. Caveats of using Minio. Caveats of using JuiceFS. The need for a VPC gateway for S3.