botocore
botocore copied to clipboard
boto
Support S3 HTTP redirects to non-Amazon URI's by using the location header
Is your feature request related to a problem? Please describe. I'm a developer contributing to Apache Ozone, an S3 compatible object store.
I would like to use HTTP redirects to balance the load amongst Ozone-based s3 servers, but botocore doesn't currently support redirects to non-amazon URI's.
Typically, http clients implement redirects using the 'location' header in the response as the destination. botocore uses the region instead of the location header . From that, it constructs an amazonaws.com based URI to redirect to. This means that botocore only ever redirects s3 requests to amazon URI's even if it made the original request to a non-Amazon server.
Describe the solution you'd like Allow botocore to support s3 redirection to non-Amazon URI's using the location header if available. This functionality would be controlled by a config variable or command line option, and would be off by default.
It could look something like this.
Thanks @GeorgeJahad for opening this feature request here for discussion. I will link the issue where this was initially brought up: https://github.com/aws/aws-cli/issues/6559
Hi @tim-finnigan: Just wondering if anyone has had time to consider this?
Hi @GeorgeJahad I don’t have any updates. Generally new feature requests will remain open for a while in order to gauge community demand before the work would be considered or prioritized by the team.
@tim-finnigan this is very important when we need to have rack-aware reads. S3 being a HTTP-based service, redirect seems like a standard operation to support.
@tim-finnigan yep this feature will be very useful. Plus it's already supported I think in the Java SDK for S3. This will just make the python lib to behave like the java one. It will be great if this PR could be accepted!
+1 @michelsumbul @GeorgeJahad @tim-finnigan. This will be a useful feature and it also keeps it compatible with Java SDK for S3.
@GeorgeJahad @tim-finnigan We've submitted PR https://github.com/boto/botocore/pull/2830 which should help Apache Ozone out here (and NVIDIA AIstore for whom I wrote it) - it would be great to see this accepted.
Hello @simontraill , @tim-finnigan , @GeorgeJahad .. Is this supported now ? I could see, still the Issue is in Open state. I was wondering, the changes could have went in as part of some other feature request/issue. Or, is there any workaround for now, so that we could use aws-cli/boto for non-AWS Server redirects ?
