virtual-host addressing style not working with custom tls verification

[MartinNowak]

Describe the bug

client = boto3.client('s3', verify=False, config=Config(s3=dict(addressing_style='virtual')))
client.list_objects_v2(Bucket='example.com', MaxKeys=1) # Endpoint provider result: https://s3.eu-central-1.amazonaws.com/example.com (path-style)

Subdomains are only allowed to use virtual-host addressing with http endpoints.

https://github.com/boto/botocore/blob/2091c2629f320c2dd16c9e29a0690d493375c8a7/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json#L5314-L5337

https://github.com/boto/botocore/blob/2091c2629f320c2dd16c9e29a0690d493375c8a7/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json#L2325-L2335

Expected Behavior

Should allow addressing subdomain buckets via virtual-host addressing style so one can properly filter egress.

Current Behavior

HTTPS endpoints are forced to use path-style addressing with subdomain buckets.

Reproduction Steps

See above.

Possible Solution

Allow virtual-host style addressing with disabled/manual TLS verification. Possibly via an additional parameter. Our workaround sets urllib3's assert_hostname to s3.amazonaws.com to still perform TLS verification. So maybe that would be a clearer parameter. Could possibly be internalised to botocore as a boolean option even.

Additional Information/Context

Important When you're using virtual-hosted–style buckets with SSL, the SSL wildcard certificate matches only buckets that do not contain dots (.). To work around this limitation, use HTTP or write your own certificate-verification logic. For more information, see Amazon S3 Path Deprecation Plan on the AWS News Blog.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

SDK version used

botocore==1.29.105

Environment details (OS name and version, etc.)

na

[tim-finnigan]

Thanks for reaching out - as mentioned in the other issue you recently opened, there are still some outstanding questions on the intended functionality here. I will mark this issue for further review by the team.

[adev-code]

Hello @MartinNowak, thanks for reaching out. I have replicated the exact same issue by using addressing_style='virtual' and a DNS named bucket, and endpoint resolution will always give a path style endpoint. Looking at how the endpoint resolution is defined, S3 actually gives this rule - not Botocore. I have reached out to the S3 Service team if this is expected. An update will be given as soon as S3 gives an update.

Internal Ref: P313943867

[adev-code]

Thank you for patience. When setting addressing_style='virtual' and using a domain name bucket (or a name that has a dot "."), the current behavior where it uses path style is an expected behavior as per the S3 Service Team. Looking at S3 Bucket Naming Rules and Best Practices, it mentions:

The following example bucket names are valid but not recommended for uses other than static website hosting because they contain periods (.): example.com www.example.com my.example.s3.bucket

Additionally:

Avoid using periods (.) in bucket names For best compatibility, we recommend that you avoid using periods (.) in bucket names, except for buckets that are used only for static website hosting. If you include periods in a bucket's name, you can't use virtual-host-style addressing over HTTPS, unless you perform your own certificate validation. The security certificates used for virtual hosting of buckets don't work for buckets with periods in their names.

This limitation doesn't affect buckets used for static website hosting, because static website hosting is available only over HTTP. For more information about virtual-host-style addressing, see Virtual hosting of general purpose buckets. For more information about static website hosting, see Hosting a static website using Amazon S3.

The Botocore follows the S3 Service Model specification:

s3.{partitionResult#dnsSuffix}/{uri_encoded_bucket}

which produces the path-style behavior you're observing. This is the expected behavior from S3 and not a bug from Botocore.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.