skyplane
skyplane copied to clipboard
[request] Support generic S3 endpoints
Please describe problem to be solved
Storage-specialized cloud providers like blackblaze and wasabi offer cheaper storage and s3 endpoints. Skyplane should
Solution
- Add programmable gateway support via #792 (in-progress)
- Support one-sided transfer plans
- Add generic S3 endpoint connector
- Support authentication with other providers (e.g. Wasabi)
Use-case One use-case a potential user described was being able to transfer 30TB between two Wasabi regions.
Another use case is transferring data to private OpenStack implementations in public HPC clusters like Compute Canada or Jetstream2. I am currently trying to export 20TB of 100 million small objects from AWS S3 to OpenStack's Object Storage API using AWS DataSync, which is taking multiple days in the "preparing" phase even with 3 agents running in parallel.