aws-cdk icon indicating copy to clipboard operation
aws-cdk copied to clipboard
verified publisher icon aws

cdk: missing ca-west-1 region in aws-entities causes cdk vended custom resources to fail deployment

Open dreamorosi opened this issue 1 year ago • 1 comments

Describe the bug

From 2.146.0 onwards, cdk vended custom resources are now region aware - this change seems to have been introduced in #30108.

To my understanding this means that when deploying a cdk vended custom resource, the LATEST_NODE_RUNTIME_MAP is used to lookup the Node.js version to use in the underlying Lambda function.

This in turn appears to look up the regions in the AWS_REGIONS_AND_RULES map in the packages/aws-cdk-lib/region-info/lib/aws-entities.ts file - which is missing an AWS region (ca-west-1).

Expected Behavior

I should be able to deploy a CDK stack that uses cdk-vended custom resources in this region.

Current Behavior

When deploying in ca-west-1, the stack fails due to the missing region in the Mapping node of the generated CloudFormation stack:

❌ Deployment failed: Error [ValidationError]: Template error: Unable to get mapping for LatestNodeRuntimeMap::ca-west-1::value

Reproduction Steps

  1. Initialize a new CDK app with npx aws-cdk@latest init app --language typescript
  2. Open lib/test-stack.ts
  3. Paste code below
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { Code, Function as LambdaFunction, Runtime } from 'aws-cdk-lib/aws-lambda';

export class TestRegionIssueStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new LambdaFunction(this, 'TestFunction', {
      runtime: Runtime.NODEJS_20_X,
      handler: 'index.handler',
      code: Code.fromInline('exports.handler = async () => "Hello, world!";'),
      logRetention: 7, // this adds a cdk vended custom resource
    });
  }
}

This will generate a cdk vended custom resource because of the logRetention prop.

  1. Run npm run cdk synth to generate the CloudFormation stack file
  2. Open the cdk.out/TestStack.template.json and find the Mappings.LatestNodeRuntimeMap
  3. Observe that the ca-west-1 region is missing from the mapping
  4. Set region to ca-west-1 via export AWS_REGION=ca-west-1
  5. Observe error during CloudFormation changeset creation
 ❌  TestStack failed: Error [ValidationError]: Template error: Unable to get mapping for LatestNodeRuntimeMap::ca-west-1::value

Possible Solution

No response

Additional Information/Context

My team Powertools for AWS Lambda deploys Lambda layers in all regions, including the missing one.

We discovered the issue in our canaries which were now failing to deploy in ca-west-1. Downgrading to 2.145.0 fixed the issue.

CDK CLI Version

2.146.0

Framework Version

No response

Node.js Version

20.x

OS

Linux, Mac

Language

TypeScript

Language Version

No response

Other information

No response

dreamorosi avatar Jun 27 '24 15:06 dreamorosi

Findings:

  • Reproducible in CDK CLI/Lib version 2.147.1 (build d3695d4). If we use the default CDK app settings without env property while creating stack,
    • It creates cdk.out/<<StackName>>.template.json with Mappings::LatestNodeRuntimeMap node with missing entry for ca-west-1 region.
    • During stack deployment, it fails during changeset creation with the below error:
      ❌  Issue30694Stack failed: Error [ValidationError]: Template error: Unable to get mapping for LatestNodeRuntimeMap::ca-west-1::value
    at Request.extractError (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:46723)
    at Request.callListeners (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91777)
    at Request.emit (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91225)
    at Request.emit (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:199828)
    at Request.transition (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:193380)
    at AcceptorStateMachine.runTo (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:158252)
    at /Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:158582
    at Request.<anonymous> (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:193672)
    at Request.<anonymous> (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:199903)
    at Request.callListeners (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91945) {
  code: 'ValidationError',
  time: 2024-06-27T20:29:50.742Z,
  requestId: '739abc90-9dc3-4d69-8435-61e8b5bb243a',
  statusCode: 400,
  retryable: false,
  retryDelay: 144.327793178868
}

 ❌ Deployment failed: Error [ValidationError]: Template error: Unable to get mapping for LatestNodeRuntimeMap::ca-west-1::value
    at Request.extractError (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:46723)
    at Request.callListeners (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91777)
    at Request.emit (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91225)
    at Request.emit (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:199828)
    at Request.transition (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:193380)
    at AcceptorStateMachine.runTo (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:158252)
    at /Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:158582
    at Request.<anonymous> (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:193672)
    at Request.<anonymous> (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:199903)
    at Request.callListeners (/Users/ashdhin/dev/repros/cdk/issue30694/node_modules/aws-cdk/lib/index.js:401:91945) {
  code: 'ValidationError',
  time: 2024-06-27T20:29:50.742Z,
  requestId: '739abc90-9dc3-4d69-8435-61e8b5bb243a',
  statusCode: 400,
  retryable: false,
  retryDelay: 144.327793178868
}

Template error: Unable to get mapping for LatestNodeRuntimeMap::ca-west-1::value
  • If we explicitly specify env property during CDK stack creation in CDK app, it does not create Mappings::LatestNodeRuntimeMap node in cdk.out/<<StackName>>.template.json and deployment succeeds.

ashishdhingra avatar Jun 27 '24 20:06 ashishdhingra

The region has been added but I'm leaving this issue open because the problem of needing to manually add regions for our custom resources to work is not yet resolved.

TheRealAmazonKendra avatar Jul 11 '24 22:07 TheRealAmazonKendra

Hi @TheRealAmazonKendra, do you know if the change with the region was already released? If no, do you know more or less when we could expect it? (<- I'm referring only to the region, not the larger effort)

For now we are pinning v2.145.0 to prevent our pipelines from stopping to work.

Thanks!

dreamorosi avatar Jul 12 '24 00:07 dreamorosi

Hi @dreamorosi the fix was merged 3 days ago so it will be released in the next version.

xazhao avatar Jul 15 '24 20:07 xazhao

Hey @dreamorosi, are you still running into this issue on the latest release? I am seeing ca-west-1 in the mapping.

scanlonp avatar Jul 17 '24 00:07 scanlonp

Hi @scanlonp - I can confirm that the issue is no longer present in version 2.149.0.

Thank you all!

dreamorosi avatar Jul 19 '24 09:07 dreamorosi

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions[bot] avatar Jul 19 '24 09:07 github-actions[bot]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

aws-cdk-automation avatar Jul 25 '24 17:07 aws-cdk-automation