skyplane icon indicating copy to clipboard operation
skyplane copied to clipboard

[request] Support generic S3 endpoints

Open sarahwooders opened this issue 2 years ago • 1 comments

Please describe problem to be solved
Storage-specialized cloud providers like blackblaze and wasabi offer cheaper storage and s3 endpoints. Skyplane should

Solution

  • Add programmable gateway support via #792 (in-progress)
  • Support one-sided transfer plans
  • Add generic S3 endpoint connector
  • Support authentication with other providers (e.g. Wasabi)

Use-case One use-case a potential user described was being able to transfer 30TB between two Wasabi regions.

sarahwooders avatar Apr 12 '23 22:04 sarahwooders

Another use case is transferring data to private OpenStack implementations in public HPC clusters like Compute Canada or Jetstream2. I am currently trying to export 20TB of 100 million small objects from AWS S3 to OpenStack's Object Storage API using AWS DataSync, which is taking multiple days in the "preparing" phase even with 3 agents running in parallel.

mihow avatar Aug 04 '23 06:08 mihow