press icon indicating copy to clipboard operation
press copied to clipboard

refactor(s3): Don't hardcode endpoint and region

Open cogk opened this issue 5 months ago • 1 comments

Don't hardcode endpoint and region for S3 buckets

There were many references to the ap-south-1 region and a global implicit reference to AWS. This PR contains refactors of the S3 bucket systems for "offsite backups", "dashboard uploads" and "Backup Bucket doctype" buckets, to avoid vendor lock-in.

I tried my best to make it backwards compatible with what I assume is the configuration on Frappe Cloud, but I can't check.

Details:

  • Added region and endpoint fields in Press Settings for the "offsite backups" and "dashboard uploads" buckets
    • Fields are: offsite_backups_endpoint, remote_uploads_region, remote_uploads_endpoint
  • Dropped unused fields: offsite_backups_provider (replaced by endpoint), data_40 (typo probably)
  • There might be some agent stuff left that is not handled correctly (e.g. retrieve the ENDPOINT_URL param here)
  • I did not add unit tests

~~cc: @gavindsouza :wave:~~ re: https://github.com/frappe/press/pull/19 https://github.com/frappe/press/pull/22


For the Frappe team

  • There was a hardcoded uploads.frappe.cloud URL somewhere, I just kept it unchanged, but the refactor won't be truly finished until a migration path can be found for you.
  • I'm not so sure about the other buckets (Backup Bucket doctype), they already had the Endpoint and Region fields, but these were not used everywhere (e.g. not in delete_s3_file)

Typical configuration values for the fields

AWS S3 (Fallback/Legacy)

  • Region: leave empty for ap-south-1
  • Endpoint: leave empty for automatic endpoint generation by boto3 = AWS S3 (I believe https://s3.amazonaws.com)

AWS S3

  • Region: ap-south-1
  • Endpoint: https://s3.ap-south-1.amazonaws.com (possibly a custom DNS name such as uploads.example.com)

OVH

  • Region: de
  • Endpoint: https://storage.de.cloud.ovh.net

cogk avatar Sep 17 '24 13:09 cogk