'cloudformation deploy' fails with S3 error

[Jarodiv]

Hello everyone.

I am working with CloudFormation in multiple regions. The process used to be pretty straight forward:

1. Create a S3 bucket -> aws s3 mb s3://my-stack-cft --profile region-1
2. Package the templates -> aws cloudformation package --profile region-1 --template-file my-stack.yaml --s3-bucket my-stack-cft --output-template-file my-stack.packed.yaml
3. Deploy the templates to multiple regions -> aws cloudformation deploy --profile region-1 --template-file my-stack.packed.yaml --stack-name my-stack -> aws cloudformation deploy --profile region-2 --template-file my-stack.packed.yaml --stack-name my-stack -> ...

This worked flawlessly, until now. When I tried to update the stack in a region, the deploy command failed with the message:

An error occurred (ValidationError) when calling the CreateChangeSet operation: S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Some debugging and it turned out that aws cloudformation deploy now seems to be unable to access CloudFormation Templates that have been uploaded by aws cloudformation package into a bucket "in a different region" (even though there shouldn't be anything like "a different region" for buckets).

Creating new buckets in all regions and uploading the templates to all of them solved the Issue but leaves a very bad taste since now all regions have to be kept in sync.

Question now is: Is this behavior intended and if so, is there any better solution (e.g. additional S3 bucket config) that restores the old behavior?

[felipeloha]

+1

[tianmarin]

Still nothing?

[kdaily]

Hi @Jarodiv @tianmarin @felipeloha, I apologize for the delay. I think this is due to this feature needing to use the newer virtual-hosted-style S3 endpoints.

https://aws.amazon.com/blogs/infrastructure-and-automation/best-practices-for-using-amazon-s3-endpoints-in-aws-cloudformation-templates/

Marking as a bug.

[inakrin]

Same issue. Deployment has begun to fail with no apparent reason and S3 error. The link referenced above provided no hint because I have no s3 references in the template.

[mikey-]

EDIT: I've updated the formatting for readability

As the error indicates, the error occurs during an attempt to create a changeset, using the specified/configured region when building the endpoint for the s3 bucket which contains the template URL. Interestingly, it seems that the error also occurs during the attempt to upload the template, but in that instance, the failed request is retried and the template is successfully uploaded.

I have encountered the same behaviour in a similar scenario:

1. create a bucket in ap-southeast-2
2. run aws --region us-east-1 cloudformation deploy --force-upload --s3-bucket bucket-in-au --s3-prefix folder-in-bucket --stack-name stack-in-us --template-file stack.yaml

Using the --debug flag reveals what occurs:

there is an attempt to upload the template via a us-east-1 endpoint:

- Calling endpoint provider with parameters: {
  'Bucket': 'bucket-in-au',
  'Region': 'us-east-1',
  'UseFIPS': False,
  'UseDualStack': False,
  'ForcePathStyle': False,
  'Accelerate': False, 
  'UseGlobalEndpoint': False,
  'DisableMultiRegionAccessPoints': False,
  'UseArnRegion': True
}
- Endpoint provider result: https://bucket-in-au.s3.us-east-1.amazonaws.com

Which results in the PermanentRedirect error:

- Received a non 100 Continue response from the server, NOT sending request body.
- https://bucket-in-au.s3.us-east-1.amazonaws.com:443 "PUT folder-in-bucket/a0b1c2d3e4f5g6h7i8j9k0lmnopqrstu.template HTTP/1.1" 301 None
- Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n
<Error>
	<Code>PermanentRedirect</Code>
	<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
	<Endpoint>bucket-in-au.s3-ap-southeast-2.amazonaws.com</Endpoint>
	<Bucket>bucket-in-au</Bucket>
	<RequestId>1234567890123456</RequestId>
	<HostId>+a0b1c2d3e4f5g6h7i8j9k0lmnopqrstuvwxyz+a0b1c2d3e4f5g6h7i8j9k0lmnopqrstuvwxyz=</HostId>
</Error>'

This error is handled well and a another attempt is made; this time using the ap-southeast-2 endpoint:

- Calling endpoint provider with parameters: {
  'Bucket': 'bucket-in-au',
  'Region': 'ap-southeast-2',
  'UseFIPS': False,
  'UseDualStack': False,
  'ForcePathStyle': False,
  'Accelerate': False,
  'UseGlobalEndpoint': False,
  'DisableMultiRegionAccessPoints': False,
  'UseArnRegion': True
}
- Endpoint provider result: https://bucket-in-au.s3.ap-southeast-2.amazonaws.com
- Updating URI from:
     https://bucket-in-au.s3.us-east-1.amazonaws.com/folder-in-bucket/a0b1c2d3e4f5g6h7i8j9k0lmnopqrstu.template
   to: 
     https://bucket-in-au.s3.ap-southeast-2.amazonaws.com/folder-in-bucket/a0b1c2d3e4f5g6h7i8j9k0lmnopqrstu.template

and we see that request go through successfully:

- 100 Continue response seen, now sending request body.
- https://bucket-in-au.s3.ap-southeast-2.amazonaws.com:443 "PUT /folder-in-bucket/a0b1c2d3e4f5g6h7i8j9k0lmnopqrstu.template HTTP/1.1" 200 0
...
- Response body:
b''

A moment later, we see a request to create a changeset using a us-east-1, path-based url for s3 location of the template:

- Event before-call.cloudformation.CreateChangeSet: calling handler <function inject_api_version_header_if_needed at 0x101a05800>
- Making request for OperationModel(name=CreateChangeSet) with params:{
  'url_path': '/',
  'query_string': '',
  'method': 'POST',
  'headers': {
    'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
    'User-Agent': 'aws-cli/2.10.3 Python/3.11.2 Darwin/22.3.0 source/arm64 prompt/off command/cloudformation.deploy'
  },
  'body': {
    'Action': 'CreateChangeSet',
    'Version': '2010-05-15',
    'ChangeSetName': 'awscli-cloudformation-package-deploy-1677804485',
    'StackName': 'stack-in-us',
    'ChangeSetType': 'CREATE',
    'Parameters.member.1.ParameterKey': 'stackParamKey',
    'Parameters.member.1.ParameterValue': 'stackParamValue',
    'Capabilities.member.1': 'CAPABILITY_NAMED_IAM',
    'Description': 'Created by AWS CLI at 2023-03-03T00:48:05.881047 UTC',
    'Tags': '',
    'TemplateURL': 'https://s3.us-east-1.amazonaws.com/bucket-in-au/a0b1c2d3e4f5g6h7i8j9k0lmnopqrstu.template'
  },
  'url': 'https://cloudformation.us-east-1.amazonaws.com/',
  'context': {
    'client_region': 'us-east-1',
    'client_config': <botocore.config.Config object at 0x10364c5d0>,
    'has_streaming_input': False,
    'auth_type': None
  }
}

which results in the error we are presented with:

- https://cloudformation.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 400 498
...
- Response body:
b'<ErrorResponse
	xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">\n  
	<Error>\n    
		<Type>Sender</Type>\n    
		<Code>ValidationError</Code>\n    
		<Message>S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.\nFor more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html</Message>\n  
	</Error>\n  
	<RequestId>01234567-0123-0123-0123-012345678901</RequestId>\n
</ErrorResponse>\n'

Seems as though all that needs to change is the CreateChangeSet request parameters, which should use the "correct" endpoint for the template url. It might also be preferable for the template url to use virtual-host-style URI.