Lambda@Edge Always Updating Without Changes

[armaneous]

What happened?

Without any changes made to the code that's deployed or the configuration, the Lambda function seems to keep registering a change requiring update which cascades to the Cloudfront Distribution requiring an update as well every time.

Example

Output of pulumi up --diff:

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:prd::infra::pulumi:pulumi:Stack::infra-prd]
    ~ aws:lambda/function:Function: (update)
        [id=<lambda-id>]
        [urn=urn:pulumi:prd::infra::aws:lambda/function:Function::<lambda-id>]
        [provider=urn:pulumi:prd::infra::pulumi:providers:<region>::<uuid>]
        code                        : archive(file:2087429) { /path/to/code.zip }
        environment                 : {
            variables : {}
        }
        handler                     : "router.handler"
        layers                      : []
        memorySize                  : 128
        name                        : "<lambda-name>"
        packageType                 : "Zip"
        publish                     : true
        reservedConcurrentExecutions: -1
        role                        : "arn:aws:iam::<account-id>:role/<role-name>"
        runtime                     : "nodejs20.x"
        skipDestroy                 : false
        tags                        : {
            environment: "..."
            project    : "..."
            service    : "..."
        }
        tagsAll                     : {
            environment: "..."
            project    : "..."
            service    : "..."
        }
        timeout                     : 3
        vpcConfig                   : {
            ipv6AllowedForDualStack: false
            securityGroupIds       : []
            subnetIds              : []
        }
    ~ aws:cloudfront/distribution:Distribution: (update)
        [id=<cf-id>]
        [urn=urn:pulumi:prd::infra::aws:cloudfront/distribution:Distribution::<cf-name>]
        [provider=urn:pulumi:prd::infra::pulumi:providers:aws::default_6_25_1::<uuid>]
      ~ defaultCacheBehavior: {
          ~ lambdaFunctionAssociations: [
              ~ [0]: {
                      ~ eventType  : "origin-request" => "origin-request"
                      + includeBody: false
                      ~ lambdaArn  : "<lambda-arn>:<lambda-version>" => output<string>
                    }
            ]
        }

Unsure if this is the problem, but the ARN is assembled using the output of the lambda resource like so:

example_lambda = lambda_.Function(...)
arn = Output.all(arn=example_lambda.arn, version=example_lambda.version).apply(lambda args: f'{args["arn"]}:{args["version"]}')

Output of pulumi about

pulumi about
CLI          
Version      3.109.0
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME           VERSION
aws            6.25.1
python         unknown
random         4.16.0
synced-folder  0.11.1

Host
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in python: executable='/home/<current-user>/.pyenv/shims/python3' version='3.11.4'

Current Stack: armaneous/infra/prd

TYPE                                                    URN
pulumi:pulumi:Stack                                     urn:pulumi:prd::infra::pulumi:pulumi:Stack::infra-prd
...
...
...


Found no pending operations associated with prd

Backend
Name           pulumi.com
URL            https://app.pulumi.com/<org>
User           <org>
Organizations  <org>
Token type     personal

Dependencies:
NAME                  VERSION
boto3                 1.34.0
gnureadline           8.1.2
pip                   24.0
pulumi_aws            6.25.1
pulumi_random         4.16.0
pulumi_synced_folder  0.11.1
setuptools            65.5.0
uuid                  1.30

Pulumi locates its logs in /tmp by default

Additional context

Not a blocker, but just a nuisance when deploying. Seems to be doing unnecessary updates.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thanks for reporting this @armaneous , I'm sorry this does not work as expected. Our team will be taking a look subject to time availability.

[mikhailshilkov]

@t0yv0 Is this another issue of diff calculation in the bridge? Should we link it to the epic then?

[t0yv0]

CC @mjeffryes I'm not sure which epic - I've been labelling these as bug/diff for the moment. On the surface it looks very much like one of those issues.

@armaneous if you have a chance to add a source program that reproduces this this could help us out a lot to make it easier to resolve. Thank you!

[arwilczek90]

I can confirm that we are seeing this as well with a function that ignores a specific tag. In our use case this is a tag that is updated at deploy time by our CI system to a version for our observability tools to read.

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

const tags = {tag1: 'value1', tag2:'value2'};

function stubNodeArchive(handler: string): pulumi.asset.AssetArchive {
  const parts = handler.split('.');
  if (parts.length !== 2) {
    throw new Error(`handler must be like "module.function"`);
  }

  const fileName = parts[0] + '.js';
  const functionName = parts[1];

  const contents: pulumi.asset.AssetMap = {};
  contents[fileName] = new pulumi.asset.StringAsset(`
          exports.${functionName} = async function(event, context) {
              console.log("EVENT: \n" + JSON.stringify(event, null, 2))
              return context.logStreamName
          };
      `);

  return new pulumi.asset.AssetArchive(contents);
}

const lambdaRole = new aws.iam.Role(
      `lambda-role`,
      {
        assumeRolePolicy: pulumi.jsonStringify({
          Version: '2012-10-17',
          Statement: [
            {
              Action: 'sts:AssumeRole',
              Effect: 'Allow',
              Principal: {
                Service: 'lambda.amazonaws.com',
              },
            },
          ],
        }),
        tags: {...tags},
      }
);
 
const _lambdaPolicyRoleAttachement = new aws.iam.RolePolicyAttachment(
      `basic-lpra`,
      {
        policyArn: 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole', // The default lambda policy for cloudwatch,etc.
        role: lambdaRole.name,
      }
 );


const lambda = new aws.lambda.Function(
      `repro-lambda`,
      {
        name: `repro-lambda`,
        code: stubNodeArchive(handler), // This is ignored after first provision because CI updates this using the aws cli, but basically just uploads a zip with a hello world for initial provisioning. You ca
        handler: 'index.handler',
        role: lambdaRole.arn,
        runtime: 'nodejs18.x',
        memorySize: 256,
        layers: [],
        timeout: 10,
        tags: {...tags}, // This is just any object of tags, just change one outside of pulumi and refresh.
      },
      { ignoreChanges: ['code', 'tags.version', 'tags_all.version', 'tagsAll.version']}
);

This accurately ignores all the tags and code updates however it also says the lambda has a 0 change update every time its run because state still sees the version tag.

[corymhall]

@armaneous do you have an example app that we can use to reproduce? I've tried to reproduce this using the below app and I was unable to. There have been some changes since you reported this issue so it is possible that it has been fixed, do you know if you still see this on the latest version?

  const role = new aws.iam.Role('role', {
    assumeRolePolicy: pulumi.jsonStringify({
      Version: '2012-10-17',
      Statement: [
        {
          Action: 'sts:AssumeRole',
          Effect: 'Allow',
          Principal: aws.iam.Principals.EdgeLambdaPrincipal,
        },
        {
          Action: 'sts:AssumeRole',
          Effect: 'Allow',
          Principal: aws.iam.Principals.LambdaPrincipal,
        },
      ],
    }),
    managedPolicyArns: [aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole],
  });

  const asset = archive.getFile({
    type: 'zip',
    sourceContentFilename: 'index.js',
    sourceContent: `
    export default function handler() {
      return 'Hello World!';
    }
    `,
    outputPath: 'lambda_function_payload.zip',
  });

  const code = new pulumi.asset.FileArchive(asset.then((a) => a.outputPath));
  const func = new aws.lambda.Function(
    'handler',
    {
      publish: true,
      role: role.arn,
      code,
      handler: 'index.handler',
      runtime: 'nodejs20.x',
    },
    {
      customTimeouts: {
        delete: '30m',
      },
    },
  );
  const bucket = new aws.s3.BucketV2('web-bucket', {});
  const id = new aws.cloudfront.OriginAccessIdentity('identity', {
    comment: 'id',
  });
  new aws.cloudfront.Distribution('distro', {
    enabled: false,
    viewerCertificate: {
      cloudfrontDefaultCertificate: true,
    },
    restrictions: {
      geoRestriction: {
        restrictionType: 'none',
      },
    },
    defaultCacheBehavior: {
      defaultTtl: 0,
      maxTtl: 0,
      cachedMethods: ['HEAD', 'GET'],
      allowedMethods: ['HEAD', 'GET'],
      targetOriginId: 'chall-origin',
      viewerProtocolPolicy: 'https-only',
      lambdaFunctionAssociations: [
        {
          eventType: 'viewer-request',
          lambdaArn: pulumi
            .all([func.arn, func.version])
            .apply(([arn, version]) => `${arn}:${version}`),
        },
      ],
      forwardedValues: {
        queryString: false,
        cookies: {
          forward: 'none',
        },
      },
    },
    origins: [
      {
        customHeaders: [
          {
            name: 'key',
            value: 'value',
          },
        ],
        originId: 'chall-origin',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 9,
        connectionAttempts: 2,
        s3OriginConfig: {
          originAccessIdentity: id.cloudfrontAccessIdentityPath,
        },
      },
    ],
  });