cassandra-medusa Support S3/GCP/Azure storage classes

Support S3/GCP/Azure storage classes

Open mohammad-aburadeh opened this issue 1 year ago • 6 comments

Medusa does not support specifying the storage class name when uploading backups to S3/GCP/Azure. This is very important for many customers as it can help to reduce the storage cost.

In my case, I manage more than 30 Cassandra clusters. We are storing more than 1000TB of backups in S3. If we store those backups in S3 CLACIER_IR storage class, we will reduce the cost to 17%.

This PR closes: #568

Mar 20 '23 09:03 mohammad-aburadeh

Hi @adejanovski

The changes are ready to review. Can someone help to review them?

Regards Mohammad

Mar 20 '23 19:03 mohammad-aburadeh

The STANDARD storage class seems to be rejected by AWS: see this test run output

It looks like the gcp tests are failing when downloading the backup (creating the backup seems to work).

Azure seems to be working, although the single part uploads aren't currently affected by the change.

Mar 24 '23 13:03 adejanovski

Hi @adejanovski

For aws upload error, I just fixed it. libcloud assumes the storage class names are lowercase. So, I just converted the class name to lowercase when passing it to libcloud.

For azure single-part upload. Unfuturnaltly, libcloud does not support specifying storage class name for Azure.

Regards Mohammad

Mar 24 '23 19:03 mohammad-aburadeh

Hi @adejanovski

I just fixed the gcp download issue. Could you please review it again and let me know?

Regards Mohammad

Mar 27 '23 19:03 mohammad-aburadeh

Kudos, SonarCloud Quality Gate passed!