miniwdl
miniwdl copied to clipboard
Support alternative S3 endpoints
The aws cli supports alternative S3 endpoints with the --endpoint-url
option. It would be great if miniwdl could expose some method of setting an alternative endpoint. I would be happy to implement this and submit a PR, but I want to do it in the right way. Specifically, I want to do it in a way that will also work with the Amazon Genomics CLI. @mlin I'd appreciate suggestions.
** Eventually this can be done much more cleanly via environment variables once https://github.com/aws/aws-sdk/pull/230 is merged.
@jdidion Thanks, for downloads this would be a straightforward modification to the downloader code here.
- The way it asks which docker image to use for AWS CLI provides an example of how to get the endpoint URL from configuration.
- Note there's a separate, similar Directory downloader below the File downloader, which we'd presumably want to modify in the same way.
- S3 downloads in AGC go through this codepath but uploads run separately -- not sure if you need one or the other or both.
- AGC uses a pinned docker image with miniwdl and miniwdl-aws baked in, so when it would be live in AGC would depend on when their devs have time/motivation to pull in a new version and release it.