(aws-cdk-lib.aws_lambda): Lambda alias/version not updated with tokenized env var

[enpatrik]

Describe the bug

When using tokenized input as a environment variable value, the Lambda alias is not automatically updated.

Expected Behavior

A new Lambda version to be created and the Lambda alias updated with the new version.

Current Behavior

The environment variables on the Lambda configuration was updated, but no new alias/version published.

Reproduction Steps

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class CdkTestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const hello = new cdk.CfnParameter(this, 'HELLO', {
      type: 'String',
    }).valueAsString

    const fn = new lambda.Function(this, 'HelloFunction', {
      runtime: lambda.Runtime.NODEJS_18_X,
      handler: "index.handler",
      code: lambda.Code.fromInline(`
        exports.handler = async function (event) {
          console.log("EVENT", event);
        };
      `),
      environment: {
        HELLO: hello
      },
    });

    fn.addAlias('live')
  }
}

# first
npx cdk deploy --parameters "CdkTest:HELLO=foo"
# second
npx cdk deploy --parameters "CdkTest:HELLO=bar"

The second time the Lambda will be updated with the new environment variable under Lambda configuration. But there won't be a new alias version published.

Same goes for other tokenized input like SSM parameter instead of CfnParameter.

When changing the value a hard coded environment variable like HELLO: 'foo' it will publish a new version when changing to HELLO: 'bar'.

Possible Solution

This might not be a bug, but since the documentation for addAlias https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda-readme.html#aliases say:

When you change your lambda code or configuration, a new resource will be created.

It's a bit unexpected behaviour when the environment variable is changed and the Lambda is updated but a new version is not created.

Additional Information/Context

There was a similar issue/feature in SAM: https://github.com/aws/serverless-application-model/issues/413

CDK CLI Version

2.84.0 (build f7c792f)

Framework Version

No response

Node.js Version

v18.16.0

OS

MacOS

Language

Typescript

Language Version

Typescript 5.0.4

Other information

No response

[peterwoodworth]

We compute the hash before the token values are known, so I'm not sure there's a way we would be able to fix this without rethinking how this construct works.

[JZechy]

@peterwoodworth @corymhall So there is the problem with my lambda deployment... While the Lambda function itself always updates with the new build tag parameter, the Alias remains unchanged. This seems to be because CDK doesn't account for late-bound values that can fundamentally alter the behavior of the Lambda function.

As a temporary workaround, I've resorted to generating a new GUID within my source code for each stack deployment. However, it would be highly beneficial if CDK could handle this natively.

What's the use case? We deploy a Lambda function from a Docker image as part of our stack, which gets updated by CodePipeline whenever there's a change in the source code. The CodeBuild project, part of the pipeline, produces not only the Docker image but also a JSON template for the stack. This template includes a parameter containing the build tag.

What are we expecting? We expect that any change in the build tag should trigger the deployment of a new version of the Lambda Alias. This is crucial for ensuring that the Alias always points to the correct version of the Lambda function.

Closing Thoughts It's reasonable to expect that any change to the Lambda function, no matter how minor, should trigger an update to its corresponding Alias. The absence of information in the documentation about the hash computation not accounting for tokens only exacerbates the confusion. This is a significant oversight that can affect production systems, and I believe it merits prompt attention.

[Lewenhaupt]

I just realised this and this makes me very worried about our production environments where it will be very difficult if we ever had to change an ssm-parameter. @peterwoodworth

[MarcBridner]

Ran into this with AWS Batch, passing the job definition ARN (that has a version) into a Lambda environment variable.

Now I'm worried what else is miswired. Large deployment, many, many stacks and resources.

[pahud]

probably related to https://github.com/aws/aws-cdk/pull/30093

[enpatrik]

The issue is still there in the latest (v2.143.0) version.

Pushed a repo with the reproduction scenario: https://github.com/enpatrik/cdk-issue-25997

[rrhodes]

Is there an ETA when this issue may be investigated further for potential fixes?

[alvazjor]

Hi all, I wanted to ask some questions to better understand how this situation is affecting your use of CDK apps.

Context on the issue

As already explained, the hashing of the CDK resources is happening before the tokenized values are known. From a CDK perspective, there is no change in the code itself (it only sees the reference to the CfnParam object, which is static, and not the value it will resolve to, that might be different).

Important considerations

1. According to the AWS Lambda documentation, modifying environment variables is an update that requires no interruption. This means that by design, changing environment variables doesn't trigger a new version creation.

2. When you deploy your CDK app with new parameter values:

   • The Lambda function's environment variables are updated immediately
   • Your function will use these new values for all subsequent invocations
   • The function continues to operate without interruption

Questions for clarification

I notice several concerns about version management, and I'd like to understand:

1. What is your specific use case for requiring new versions when environment variables change?
2. Are you using the versions/aliases for traffic shifting or rollback scenarios?
3. Is there a specific compliance or audit requirement that necessitates version tracking for environment variable changes?

This information would help us better understand the impact and potentially explore alternative solutions or workarounds that might better suit your needs.

Current status

While this behavior might seem unexpected, it's important to note that the Lambda function is still operating with the updated environment variables, even though a new version isn't created. If version tracking is critical for your use case, we could explore alternative patterns or workarounds while we evaluate potential changes to the current behavior.

[enpatrik]

Hi @alvazjor ! Thanks for taking the time to look into this and adding the additional information, much appreciated!

Answers to your questions:

1. One example would be if the value of LOG_LEVEL (used by our log library in the Lambda code) was changed via CfnParam.
2. We use .addAlias('live') for integrations such as APIGW->Lambda, to prevent issues as described by Aidan Steele
3. No, not for us anyway.

Maybe there's some alternative approach to this? But right now the only option I know is to skip either live-alias or tokenized env variables.

[aemada-aws]

Potential related/duplicate issues:

• https://github.com/aws/aws-cdk/issues/27889
• https://github.com/aws/aws-cdk/issues/11433