Question: How to properly configure cross-region deployment

[engrun]

Background: Installed ADF (3.0.6) about 6 months ago, and have been deploying stuff to one-region only without issues. adfconfig.yml was configured with eu-west-1 as deployment region and targets was also set to eu-west-1.

regions:
  deployment-account: eu-west-1 # The region you define as your main deployment region
  targets: # A list of regions you wish to bootstrap and also deploy into via pipelines
    - eu-west-1

Problem: Today I was tasked with setting up a pipeline for deploying cost anomaly monitors, and since this is a regional CFN resource type, I need to deploy this stack to us-east-1.

So after reading up on the docs, I re-configured the regions to

regions:
  deployment-account: eu-west-1 
  targets: 
    - us-east-1

This creates a regional deployment bucket and kms key.

So far so good.

I then defined the pipeline in the deployment maps with the following to override the region.

   targets:
      - path: 123456789 #my-test-account
        regions: us-east-1

The following then happens

• The pipeline is created, and triggered.
• Codebuild does it magic and uploads the template/artifact to the regional bucket
• Boom! cloudformation deploy fails with an AccessDenied (and btw, it does not say why!)

So after quite a bit of digging we found two issues

1. The adf-cloudformation-role in the destination account has only read-access to the main region deployment bucket, not to the regional deployment bucket
2. The adf-cloudformation-role in the destination account does not have access to use the regional kms key

So after manually updating the role-policy in the console, and adding permissions ( see ADDED: below), the pipeline works as expected and cloudformation successfully deploys a stack in the us-east-1 region

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*"
            ],
            "Resource": [
                "arn:aws:s3:::adf-main-region-bucket",
                "arn:aws:s3:::adf-main-region-bucket/*",
                "ADDED: arn:aws:s3:::adf-new-region-bucket",
                "ADDED: arn:aws:s3:::adf-new-region-bucket/*"
            ],
            "Effect": "Allow",
            "Sid": "S3"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:Encrypt",
                "kms:GenerateDataKey*",
                "kms:ReEncrypt*"
            ],
            "Resource": [
                "arn:aws:kms:main-region-key",
                "ADDED: arn:aws:kms:new-region-key"
            ],
            "Effect": "Allow",
            "Sid": "KMS"
        }
    ]
}

So my question comes down to:

Given that IAM resources are global, and the roles has this defined in the global.yml file

- !Sub arn:aws:s3:::${DeploymentAccountBucketName}
- !Sub arn:aws:s3:::${DeploymentAccountBucketName}/*

this means that the role is only granted access to the bucket in the main region. Also, Parameter Store is populated with bucket_name and kms_arn in the new region. But what is the point of this, as these are not used?

Are you required to manually modify these policies yourself?

Anyways - info on how to properly configure for multiple regions would be very helpful :)

[engrun]

My current "workaround" is as follows

• Roll out a stack for all accounts in the organization, adding two new SSM params in the main region

  • /adf/cross_region/kms_arn/us-east-1
  • /adf/cross_region/s3_regional_bucket/us-east-1

• Then updating the adf-cloudformation-role-policy in global.yml to include:

- Effect: Allow
            Sid: "S3"
            Action:
              - s3:Get*
              - ..
            Resource:
              - !Sub arn:aws:s3:::${DeploymentAccountBucketName}
              - !Sub arn:aws:s3:::${DeploymentAccountBucketName}/*
              - !Sub arn:aws:s3:::${DeploymentAccountUsEast1RegionBucketName}
              - !Sub arn:aws:s3:::${DeploymentAccountUsEast1RegionBucketName}/*
          - Effect: Allow
            Sid: "KMS"
            Action:
              - kms:Decrypt
              - ..
            Resource:
              - !Ref KMSArn
              - !Ref KMSUsEast1Arn

where $DeploymentAccountUsEast1RegionBucketName and $KMSUsEast1Arn are then fetched from SSM Param Store

[engrun]

Update, this workaround causes problems when creating new accounts.

The bootstrap-pipeline fails with

botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateChangeSet operation: Unable to fetch parameters [/adf/cross_region/kms_arn/us-east-1,/adf/cross_region/s3_regional_bucket/us-east-1] from parameter store for this account.

This is indeed negative side-effect of modifying the adf-cloudformation-role policy ( as described above ).

So, the problem comes down to (again) that ADF is not properly bootstrapping regions.

There is no recognition from the ADF team on this issue, but it would be nice to know if this issue is fixed in 3.1.x versions.

[kalleeh]

Hi, I'm on ADF 3.1.2 and tried re-creating your problem but it worked for me. I don't think 3.0.6 should have any differences in this particular aspect however since multi-region has been existing for quite a while.

The only differences I see is that I'm on 3.1.2, and I've also modified adfconfig.yml by adding the target region to the list rather than replacing it.

regions:
  deployment-account: eu-west-1 
  targets: 
    - eu-west-1
    - us-east-1

Maybe this is something you can test? I'll try adding a 3rd region in the same way and see if I run into any problems.

[tharindu-qoot]

Hi, I'm having this same issue. I'm running ADF_VERSION: 3.1.2. Can someone please confirm the solution posted by Kalleeh above is the way to solve this?

Thanks.

[tharindu-qoot]

FYI, that did not work. @engrun please let me know if you managed to find a solution to this.

Thanks.

[engrun]

@tharindu-qoot The workaround I described above worked for me. In the near future we'll need to configure a new region, so we'll test this again then

[tharindu-qoot]

Thanks @engrun. I ended up doing something similar.. I'm deploying a regional.yml template for my workload OU. Something like below, which will attach an inline policy to the 'adf-cloudformation-role

AWSTemplateFormatVersion: "2010-09-09"
Description: Attach an inline policy to adf-cloudformation-role, to allow access to regional deployment resources.

Parameters:
  KMSArn:
    Type: "AWS::SSM::Parameter::Value<String>"
    Description: ARN of the KMS CMK created in the Deployment account
    Default: kms_arn
  DeploymentAccountBucketName:
    Type: "AWS::SSM::Parameter::Value<String>"
    Description: Deployment Bucket Name
    Default: bucket_name
  DeploymentAccountId:
    Type: String
    Description: Deployment Account ID
    Default: '1234567890'

Resources:
  AllowCFNRoleAccessToRegionalKmsS3:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: !Sub 'allow-regional-kms-s3-access-${AWS::Region}'
      Roles:
        - 'adf-cloudformation-role'
        - 'adf-cloudformation-deployment-role'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Action:
          - s3:Get*
          - s3:List*
          - s3:Put*
          Resource:
          - !Sub 'arn:aws:s3:::${DeploymentAccountBucketName}'
          - !Sub 'arn:aws:s3:::${DeploymentAccountBucketName}/*'
          Effect: Allow
          Sid: S3
        - Action:
          - kms:Decrypt
          - kms:DescribeKey
          - kms:Encrypt
          - kms:GenerateDataKey*
          - kms:ReEncrypt*
          Resource: !Ref KMSArn
          Effect: Allow
          Sid: KMS

This works, Cfn can deploy the template from S3.

But then I ended up with more questions, cause the adf-cloudformation-deployment-role doesn't have any permissions to execute the changeset, as in it can't create any resources. So now I'm attaching more policies to that role.

This doesn't seem right, were you able to deploy stacks cross-region ?

[engrun]

This doesn't seem right, were you able to deploy stacks cross-region ?

Yes. However, when you are bootstrapping a new region, ADF will create a new deployment-bucket for that region, named something like adf-regional-base-deploy-deploymentframe....

So in your policy you need to reference this bucket instead. The same goes for KMS.

As you can see in my workaround above, I am referencing these specifically

 - !Sub arn:aws:s3:::${DeploymentAccountUsEast1RegionBucketName}
 - !Sub arn:aws:s3:::${DeploymentAccountUsEast1RegionBucketName}/*

and

 - !Ref KMSUsEast1Arn

[sbkok]

I'm closing this issue as it has been inactive for a long time. This probably means that it is not reproducible or it has been fixed in the meanwhile.

Please reopen if you still encounter this issue with the latest stable version.

Thank you!

[StewartW]

Adding an additional comment on this:

This issue typically occurs when the "Enable Cross Account Access" state machine fails in your deployment account. If you see this issue, I'd suggest checking the status of that first.