miniwdl icon indicating copy to clipboard operation
miniwdl copied to clipboard

Support alternative S3 endpoints

Open jdidion opened this issue 2 years ago • 1 comments

The aws cli supports alternative S3 endpoints with the --endpoint-url option. It would be great if miniwdl could expose some method of setting an alternative endpoint. I would be happy to implement this and submit a PR, but I want to do it in the right way. Specifically, I want to do it in a way that will also work with the Amazon Genomics CLI. @mlin I'd appreciate suggestions.

** Eventually this can be done much more cleanly via environment variables once https://github.com/aws/aws-sdk/pull/230 is merged.

jdidion avatar Aug 31 '22 21:08 jdidion

@jdidion Thanks, for downloads this would be a straightforward modification to the downloader code here.

  1. The way it asks which docker image to use for AWS CLI provides an example of how to get the endpoint URL from configuration.
  2. Note there's a separate, similar Directory downloader below the File downloader, which we'd presumably want to modify in the same way.
  3. S3 downloads in AGC go through this codepath but uploads run separately -- not sure if you need one or the other or both.
  4. AGC uses a pinned docker image with miniwdl and miniwdl-aws baked in, so when it would be live in AGC would depend on when their devs have time/motivation to pull in a new version and release it.

mlin avatar Sep 01 '22 04:09 mlin