Error message for `aws s3 ls` refers to wrong endpoint.

[wonko]

Describe the bug

Given an s3 bucket in eu-west-1, when setting AWS_REGION=eu-west-1 and AWS_ENDPOINT_URL=https://s3.amazonaws.com env vars and issuing an aws s3 ls s3://bucketname/some/path/to/object, it will give you the following error message:

An error occurred (PermanentRedirect) when calling the ListObjectsV2 operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: bucketname.s3.amazonaws.com
You can fix this issue by explicitly providing the correct region location using the --region argument, the AWS_DEFAULT_REGION environment variable, or the region variable in the AWS CLI configuration file.  You can get the bucket's location by running "aws s3api get-bucket-location --bucket BUCKET".

It is right that it gives the error, as the endpoint should either be unset, or set to the region-specific s3.eu-west-1.amazonaws.com endpoint. The endpoint which is suggested in the error message is however wrong, and will result in NotFound errors.

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

The error message should either give a link to how/where to find the correct endpoint, or it should give the correct endpoint. Giving a non-functional endpoint is a bug to me.

Current Behavior

See the output in the above snippet

Reproduction Steps

Create a new bucket in non us-east-1, make sure you have access to it, and then:

bash-5.1$ set | grep AWS
AWS_ENDPOINT_URL=https://s3.amazonaws.com
AWS_REGION=eu-west-1
AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile
_=AWS_ENDPOINT_URL=https://s3.amazonaws.com
bash-5.1$ aws s3 ls s3://[BUCKET-REDACTED]/some/path/ --summarize

An error occurred (PermanentRedirect) when calling the ListObjectsV2 operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: [BUCKET-REDACTED].s3.amazonaws.com
You can fix this issue by explicitly providing the correct region location using the --region argument, the AWS_DEFAULT_REGION environment variable, or the region variable in the AWS CLI configuration file.  You can get the bucket's location by running "aws s3api get-bucket-location --bucket BUCKET".
bash-5.1$ export AWS_ENDPOINT_URL="https://[BUCKET-REDACTED].s3.amazonaws.com"
bash-5.1$ aws s3 ls s3://[BUCKET-REDACTED]/some/path/ --summarize

An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist.
bash-5.1$ unset AWS_ENDPOINT_URL
bash-5.1$ aws s3 ls s3://[BUCKET_REDACTED]/some/path/ --summarize

Total Objects: 0
   Total Size: 0

(it also works when setting the correct endpoint s3.eu-west-1.amazonaws.com)

Possible Solution

Change the error message.

Additional Information/Context

No response

CLI version used

latest

Environment details (OS name and version, etc.)

ubuntu docker container, latest, amd64

[adev-code]

Hello @wonko, thanks for reaching out. I have replicated the same issue and followed the error and still lead to an error. Adding the actual region of the bucket like export AWS_ENDPOINT_URL=https://s3.eu-west-1.amazonaws.com actually solves the issue. Thank you for seeing this. I have reached out to the S3 Service Team in regards. I will update if there are any updates from them. If you have any questions, please let me know. Thanks.

For Internal Tracking : P245495726

[adev-code]

Hello @wonko, thanks for the patience. Revising the issue, I have found that when putting AWS_ENDPOINT_URL=https://s3.amazonaws.com , it takes precedence for the host URL to be used. Since AWS_ENDPOINT_URL is passed, then 'ForcePathStyle' is set to True 'ForcePathStyle': True. This means that the request to S3 would be https://s3.amazonaws.com/[bucketname] rather than the virtual path of https://bucket-name.s3.region-code.amazonaws.com.

The reason why ForcePathStyle is set to true if there is an AWS_ENDPOINT_URL passed is for compatibility. When you specify a custom endpoint URL, the CLI doesn't know if that endpoint supports virtual-hosted style addressing (which requires DNS configuration for each bucket name). Path-style addressing is more universally compatible, so it's used by default.

Without having AWS_ENDPOINT_URL, the CLI by default uses virtual path to S3. Although, could you please clarify why you would need to use AWS_ENDPOINT_URL=https://s3.amazonaws.com ?

If you would still want to use AWS_ENDPOINT_URL=https://s3.amazonaws.com , then please ensure that you enable Virtual Path too by entering the command :

$ aws configure set s3.addressing_style virtual

and re run your S3 command.

Please let me know if you have any questions. Thank you.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[wonko]

The issue I had was that the values are set by some external tooling/scripting for taking backups. Their logic (see https://github.com/percona/percona-docker/blob/main/percona-xtradb-cluster-8.4-backup/lib/pxc/aws.sh#L6-L7) was to set the default value of https://s3.amazonaws.com if no endpoint was specified (instead of leaving it empty, so the SDK would determine it based on the region). So that's a small bug imho on their side, there's an issue open there for this.

But, the main issue i wanted addressing in this ticket is just the wording of the error message. I got the config for the SDK wrong (unwillingly, in my case, but I know what an endpoint and a region is), get a recommendation to use a different endpoint, and the recommendation doesn't work, giving an other error.

I guess this should be addressed by either taking away the recommendation and maybe adding a generic response (maybe including a link towards the docs for s3 endpoints). It might help other people in the same case, not having to scratch their heads. Even just outputting the settings which matter, like "you configured endpoint xxx, region yyy and setting ABC to zzz, which probably won't work. Visit xyz for more info on s3 endpoints" should give better guidance.