codepipeline-actions: CloudFormationCreateUpdateStackAction fails when lambda requires assets

[traysonkelii]

Describe the issue

I have a few stacks (ServiceStack,AuthenticationStack, and DataStorageStack). All of which work when deployed locally from the CLI. I decided to try and create a simple CICD pipeline using thePipeline construct from the aws-cdk-lib/aws-codepipeline module. A simple github source to main, along with a simple build step (npm ci, cdk synth). I then use the cdk.out generated in an artifact to perform actions specifically the CloudFormationCreateUpdateStackAction on the stacks (essentially update them). Below is the code for the PipelineStack:

export class PipelineStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const pipeline = new Pipeline(this, "Pipeline", {
      pipelineName: "CombatSportsRankingPipeline",
      crossAccountKeys: false,
    });

    const sourceOutput = new Artifact("sourceOutput");

    pipeline.addStage({
      stageName: "Source",
      actions: [
        new GitHubSourceAction({
          owner: "XXXX",
          repo: "XXXX",
          branch: "main",
          actionName: "Pipeline_Source",
          oauthToken: SecretValue.secretsManager(
            "XXXX"
          ),
          output: sourceOutput,
        }),
      ],
    });

    const codeBuildOutput = new Artifact("codeBuildOutput");

    pipeline.addStage({
      stageName: "Build",
      actions: [
        new CodeBuildAction({
          actionName: "Code_Build",
          input: sourceOutput,
          outputs: [codeBuildOutput],
          project: new PipelineProject(this, "CodeBuildProject", {
            environment: {
              buildImage: LinuxBuildImage.STANDARD_7_0,
            },
            buildSpec: BuildSpec.fromSourceFilename(
              "build-specs/code-build.yml"
            ),
          }),
        }),
      ],
    });

    pipeline.addStage({
      stageName: "Pipeline_Update",
      actions: [
        new CloudFormationCreateUpdateStackAction({
          actionName: "Pipeline_Update",
          stackName: "PipelineStack",
          templatePath: codeBuildOutput.atPath("PipelineStack.template.json"),
          adminPermissions: true,
        }),
      ],
    });

    pipeline.addStage({
      stageName: "DataStorage_Update",
      actions: [
        new CloudFormationCreateUpdateStackAction({
          actionName: "DataStorage_Update",
          stackName: "DataStorageStack",
          templatePath: codeBuildOutput.atPath(
            "DataStorageStack.template.json"
          ),
          adminPermissions: true,
        }),
      ],
    });

    pipeline.addStage({
      stageName: "Authorization_Update",
      actions: [
        new CloudFormationCreateUpdateStackAction({
          actionName: "Authorization_Update",
          stackName: "AuthorizationStack",
          templatePath: codeBuildOutput.atPath(
            "AuthorizationStack.template.json"
          ),
          adminPermissions: true,
        }),
      ],
    });

    pipeline.addStage({
      stageName: "CsrService_Update",
      actions: [
        new CloudFormationCreateUpdateStackAction({
          actionName: "CsrService_Update",
          stackName: "CsrServiceStack",
          templatePath: codeBuildOutput.atPath("CsrServiceStack.template.json"),
          adminPermissions: true,
        }),
      ],
    });
  }
}

The Authorization_Update stage fails with the error:

Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400)"

I did check the S3 bucket and the asset.zip file found in the template was there. I don't know if there is any other steps I need to do on my part.

I checked the documentation and couldn't find anywhere that suggested whether I should enabled any pipeline service principles access to the lambdas or if there were other parameters (selfMutating doesn't seem to be an option in this v2 version of code pipelines). I may be lost a little as to what this entails.

I've looked through codepipeline module and the CloudFormationCreateUpdateStackAction which seemed extra lean. Any guidance is appreciated.

Links

• https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codepipeline_actions.CloudFormationCreateUpdateStackAction.html
• https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codepipeline_actions-readme.html
• https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codepipeline-readme.html

[pahud]

I was not able to deploy with the provided code snippets. Are you able to simplify it and provide all necessary info including the buildspec of codebuild and a sample source repo so we can simply run in our local environment and see what's happening?

[traysonkelii]

Here is the buildspec, I will need to work on the sample source repo:

version: 0.2

phases:
  install:
    commands:
      - npm install -g npm
      - npm install
  build:
    commands:
      - npm run clean
      - npm run build
      - npm run cdk -- synth

artifacts:
  base-directory: cdk.out
  files:
    - "**/*"

Here is the actual github repo (it should be public: https://github.com/traysonkelii/combat-sports-ranking-cdk) I'll work on making it smaller.

[seat-uat]

+1 and bump , Also facing same issue

[DimitriosKay]

Have the exact same issue in a very similar setup.

I am however setting the Artifacts target bucket in aws-codepipeline.Pipeline

    this.deployPipeline = new Pipeline(this, 'Pipeline', {
      artifactBucket: artifactsS3,
    });

When looking through what this all generates, I see (in artifactsS3) the output .zip after the Build stage, just as expected. It has the cdk.out directory with stacks and assets that match the object keys of each other. In assets and template, the destination bucketName is cdk-XXX-assets-ACCOUNT-REGION, but the referred zip and json files are not found in there when having a look. Is this a hint that when calling CloudFormationCreateUpdateStackAction in the Deploy Stage with templatePath: codeBuildOutput.atPath("STACKNAME.template.json"), it does not look at the Input Artefact from the Build Output, but instead looks at cdk-XXX-assets-ACCOUNT-REGION/assetIDNumber.zip

Note: If I deploy the same bundle of stacks (Lambda function and Pipeline linking to deployed Lambda function) it works perfectly fine. No changes other than naming of stacks.

[DimitriosKay]

Solved my issue. I'll try to elaborate on what led me to the decision that fixed it.

TLDR I redeployed both the Lambda stack and the Pipeline stack. (The Lambda stack is the one that the Pipeline stack updates with CloudFormationCreateUpdateStackAction)

I was working from two branches: testing and master Originally the Lambda stack was deployed from testing (with a local cdk deploy). Later on I merged my changes into master. When the Pipeline ran, the Source step pulls from master, then Build does its synth as expected, the assets are not found in the cdk S3 bucket and the Deploy fails with NoSuchKey ... Sad. Checking the Build logs though, there was a mismatch between what the Lambda stack asset ID was (in CodeBuild) and what it was in my local cdk.out directory. That made me think that whatever I deployed before was somehow seen as different (but not too different) to what the Pipeline was trying to update later, forcing it to create a new asset. Changed my branch to master, did a synth and the local asset ID matched the one CodeBuild generated.

If anyone can shed more low level light on this, I'd be all ears.