CreateTransitGatewayVpcAttachment failing with Client.UnauthorizedOperation in spoke account

[kbosco]

TGW attachment creation is failing in spoke account due to an access denied error. Haven't changed any templates and all the roles seems have the required permissions and trust relationships

{ ...... "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "", "arn": "arn:aws:iam::*:role/TransitNetworkExecutionRole-us-east-1", "accountId": "", "userName": "TransitNetworkExecutionRole-us-east-1" }, ....., "eventTime": "", "eventSource": "ec2.amazonaws.com", "eventName": "CreateTransitGatewayVpcAttachment", "awsRegion": "us-east-1", "sourceIPAddress": "", "userAgent": "", "errorCode": "Client.UnauthorizedOperation", "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message: *, ..... }

[avenging]

Hi @kbosco

I've had this recently, I think AWS have changed something. Had to change the policy in the Spoke template to add the Spoke account number as well as the Hub for the transit gateway ARN.

- !Sub arn:aws:ec2:${AWS::Region}:${HubAccount}:transit-gateway/*
- !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/*

[groverlalit]

Thanks for bring this to our attention. We will try to duplicate the issue as well.

@kbosco Can you please confirm that the policy snippet found here is causing "UnauthorizedOperation" error.

@avenging @kbosco Can you also confirm that adding the resource below for 'CreateTransitGatewayVpcAttachment' action fix the issue? '!Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/*'

Thanks.

[kbosco]

Thanks for bring this to our attention. We will try to duplicate the issue as well.

@kbosco Can you please confirm that the policy snippet found here is causing "UnauthorizedOperation" error.

@avenging @kbosco Can you also confirm that adding the resource below for 'CreateTransitGatewayVpcAttachment' action fix the issue? '!Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/*'

Thanks.

The recommendation from @avenging did resolve the "UnauthorizedOperation" error.

[avenging]

Yes adding '!Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/*' Fixed the issue for me.

[groverlalit]

@avenging Thanks for the confirmation. :)

@kbosco @avenging Can you please confirm if this happening in a particular region or multiple regions?

[kevinjamescasey]

It is happening in Frankfurt. I have not checked other regions. I find it strange that the resource ARN for the transit gateway in the decoded error message includes the spoke account ID and instead of the specific transit gateway ID it has a wildcard. I agree with @avenging that AWS must have changed something with the CreateTransitGatewayVpcAttachment action.

[avenging]

@kevinjamescasey @groverlalit Yes checking the decoded error is how I realised what the issue was. I had a script re-creating vpc attachments across multiple regions, off the top of my head eu-central-1, us-east-1 and us-west-2. I think 2 of the 3 failed and one worked which was one of the US regions, which I think was us-west-2. Sorry over a week ago now.

[chrusty]

I've also experienced this today in ap-southeast-2. Had me scratching my head for a while! I tried @avenging solution, which solved it for that specific spoke account.

I'd love to know what changed to cause this. STNO isn't exactly the easiest thing to debug!

[bltb]

It broke for us too in ap-southeast-2. We noticed this yesterday. We see the same decoded authorization message weirdness as @kevinjamescasey

[groverlalit]

We have escalated this to the TGW service team and they are working on it. Thanks to @avenging for sharing the workaround with us. Also, we do not recommend to replace the account in the ARN as the workaround to avoid rolling back the changes.

[bltb]

We have escalated this to the TGW service team and they are working on it. Thanks to @avenging for sharing the workaround with us. Also, we do not recommend to replace the account in the ARN as the workaround to avoid rolling back the changes.

@groverlalit , we heard from AWS Premium Support - who heard from the Transit Gateway service team (who you must have contacted) - that this unintentional change has been fixed / reverted. They said it was fixed on March 26th 2021 at 3:30PM PDT.