S3_IGNORE_SUBDOMAIN_BUCKETNAME undocumented + AWS_ENDPOINT_URL gotcha

[jcrben]

TL;DR - some additional guidance on the two gotchas that I hit in the past couple days could be added to the docs. I can try to take a stab at some language if you think that makes sense

I've been using moto for a fair bit, but lately started using the server mode for integration tests a bit more deeply.

This reproduction case is run on Ubuntu 22.04. I installed moto like this:

pipx install moto[server]

I launched the server with moto_server in one terminal.

In another, I ran this with aws 2.15.27

export AWS_ENDPOINT_URL=http://localhost:5000
aws cloudformation deploy --debug --template-file ./test1.yaml --s3-bucket test-bucket --s3-prefix test-prefix --stack-name test-stack --region us-east-1

using this yaml:

AWSTemplateFormatVersion: 2010-09-09
Description: dummy test
Resources:
    RandomSSMParameter:
        Type: AWS::SSM::Parameter
        Properties:
        Name: /my/test
        Type: String
        Value: "simple string"

I got the following error:

botocore.exceptions.ClientError: An error occurred (Unknown) when calling the CreateChangeSet operation: Unknown

An error occurred (Unknown) when calling the CreateChangeSet operation: Unknown

I initially noticed a NoSuchBucket which led me to https://github.com/getmoto/moto/issues/2915 where I learned about S3_IGNORE_SUBDOMAIN_BUCKETNAME. I restarted the server with that, which resolved NoSuchBucket error but it still failed.

After stepping thru some code and poking around at issues a bit, I retried the command with this instead: export AWS_ENDPOINT_URL=http://localhost:5000

That worked.

I originally used 127.0.0.1 instead of localhost as I watched https://www.youtube.com/watch?v=98SYTvNw1kw because I've been hit by localhost resolving to an ipv6 before and I figured I'd cut that out, but I can understand why dealing with the IP address can be challenging in some cases.

Anyway, I did try to review the docs carefully as I hit this issue, and in looking back over them I found that there's nothing about S3_IGNORE_SUBDOMAIN_BUCKETNAME https://docs.getmoto.org/en/latest/search.html?q=S3_IGNORE_SUBDOMAIN_BUCKETNAME&check_keywords=yes&area=default#

The docs over at https://docs.getmoto.org/en/latest/docs/server_mode.html do use localhost:5000 but it might be worth pointing out that 127.0.0.1:5000 is not an adequate replacement.

[bblommers]

Hi @jcrben, it would indeed be nice to have both mentioned in the docs.

The logical place for the environment variable would probably the S3 docs themselves: https://github.com/getmoto/moto/blob/33d374526303f9b3b686402a2074963dd027fa65/moto/s3/models.py#L1616

Any change there will automatically show up in http://docs.getmoto.org/en/latest/docs/services/s3.html

I'm not sure what the best place is to mention the 127.0.0.1 gotcha, so I'm open to suggestions there.