s3 head_object on different client region triggering multiple retries to get bucket region

[thehesiod]

We have a report in aiobotocore that the head_object requests are very slow, I've traced this back to inefficiency in botocore when performing head_object with default sig algo which doesn't require a region and bucket is in a different region than the client. The problem is exacerbated since each 400 returns a Connection: close which causes a new connection to AWS and connections are more expensive in aiohttp.

Here's what I found after doing a head_object request:

1. (r1, initial, Connection: closed) 400 on initial region-less head_object with key
2. (r2, failed get region, Connection: closed) this emits needs-retry.s3.HeadObject, which calls S3RegionRedirector.redirect_from_error which calls utils.py:get_bucket_region which triggers another request which also returns a 400 since it's still in the wrong region.
3. (r3, retry get region, not closed) this then triggers another 'needs-retry.s3.HeadBucket', which emits the same event and calls the same redirect_from_error which returns a 0 and causes the request to be retried, again since it's not None.
4. this time we get a 200 because the host is set to the correct region, so we unwind back to the get_bucket_region call.
5. we now unwind back to the needs_retry from the initial HeadObject which returns True
6. (r4, initial retry, not closed) do the initial HeadObject again with the region set and it succeeds.

So basically one head_object request results in 4 requests to AWS over 3 connections. It sounds like botocore is optimized to expect the bucket to be in the same region, vs

Steps to reproduce head_object on s3 key in bucket in different region than client, which defaults to us-east-1

Expected behavior I think low hanging fruit is that r2 already has the bucket name in the header: x-amz-bucket-region: us-west-2, so there seems to be no reason to retry it even though it gave a 400, this will result in one fewer connections.

Ideally No connections would need to be tossed, however that would require like a param to head_object to tell it to first retrieve the bucket location before attempting to do the head_object. I supposed this can be delegated to client behavior as well

anyways I think the minor optimization would be a good one, can probably just tell the bucket region to not do a retry

[stobrien89]

Hi @thehesiod,

Thanks for pointing this out! I'll pass along to the team. Marking as an enhancement for now.