Update S3 Server "region" along with the "endpoint_url"

[SoniaComp]

Describe the bug

When I use MinioServer as the boto3 endpoint, if my "bucket region" is set incorrectly, the "s3 host" is also changed along with the "region" update.

Expected Behavior

When I use MinioServer as the boto3 endpoint, if my "bucket region" is set incorrectly, the "s3 host" does not change and only the "bucket region" is updated.

Current Behavior

If my "bucket region" is set incorrectly, botocore update the correct "region", but change the "s3 host" to "amazonaws.com".

Reproduction Steps

python code

import boto3
import os
from urllib.parse import urlparse
from botocore.config import Config

# set s3_client with wrong region 
# if your s3_server region is different from default region(us-east-1), you do not need to specify.
s3_client = boto3.client(
    service_name='s3',
    aws_access_key_id='${aws_key}',
    aws_secret_access_key='${aws_secret}',
    endpoint_url='${minio_server_url:port}', 
)

boto3.set_stream_logger(name='botocore')

s3_client.list_buckets() # works well
s3.list_objects(Bucket='${bucket_name}', Prefix='${object_key}')) # update region and change host_endpoint to "amazonaws.com"

debug log

$ botocore.hooks [DEBUG] Event needs-retry.s3.ListObjects: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8ee1ed82b0>>
$ botocore.utils [DEBUG] S3 client configured for region us-east-1 but the bucket ${bucket_name} is in region ap-northeast-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
$ botocore.utils [DEBUG] Updating URI from http://{minio_server_url:port}/${bucket_name}?prefix=${object_key}&encoding-type=url to http://s3.ap-northeast-2.amazonaws.com/${bucket_name}?prefix=${object_key}&encoding-type=url

Possible Solution

It seems that it can be fixed by changing the endpoint parsing logic. I would like to contribute by creating a PR within this week.

Additional Information/Context

minio server using docker image "quay.io/minio/minio:RELEASE.2022-05-08T23-50-31Z"

SDK version used

1.24.38

Environment details (OS name and version, etc.)

CentOS Linux release 8.3.2011, MacOS 12.2.1

[tim-finnigan]

Hi @SoniaComp thanks for reaching out. I'm not sure if we can guarantee compatibility with third-party services like Minio when it comes to this type of behavior. And any endpoint-related changes would likely need to happen at a cross-SDK level rather than only directly in boto3.