Have to apply twice when lambda function build changes

[cahnk]

trafficstars

Description

When the function associated with our Lambda must be rebuilt we have to execute apply twice. The first time fails with the error message listed below. The second apply is successful. We are using a local /src directory for our Python code. This behavior also occurs the first time we run apply after implementing the Lambda module. This appears to be a dependency issue.

• [x] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• 3.3.1

• Terraform version:

terraform {
  required_version = "1.2.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23.0"
    }
  }
}

Reproduction Code [Required]

Steps to reproduce the behavior:

Are you using workspaces? No Have you cleared the local cache (see Notice section above)? Yes

List steps in order that led up to the issue you encountered

1. Modify the internal build
2. Run apply
3. Receive errors shown below
4. Rerun apply
5. Apply runs successfully

Expected behavior

The apply should run successfully on the first attempt.

Actual behavior

It requires two attempts for the apply to complete successfully.

Terminal Output Screenshot(s)

Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ module.lambda_inert_fanout.module.lambda.aws_lambda_function.this[0] to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .source_code_hash: was
│ cty.StringVal("6SxcGdAfmMFk95b4cpr0+YJKFKwk35qic4W4mv6xUEc="), but now
│ cty.StringVal("v3qRfvFuSaKzQslbsNO1Rc4HOZuFwvExGpLRO1utnrk=").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: local-exec provisioner error
│ 
│   with module.lambda_inert_calculator.module.lambda.null_resource.archive[0],
│   on .terraform/modules/lambda_inert_calculator.lambda/package.tf line 65, in resource "null_resource" "archive":
│   65:   provisioner "local-exec" {
│ 
│ Error running command
│ 'builds/c458b42518ccdd1280b4301c65c5cb9ad4204df92543efadf0a7e8108461976e.plan.json':
│ exit status 1. Output: zip: creating
│ 'builds/c458b42518ccdd1280b4301c65c5cb9ad4204df92543efadf0a7e8108461976e.zip'
│ archive
│ zip: adding content of directory: ./src
│ zip: adding: .terragrunt-source-manifest
│ zip: adding: inert_fanout.py
│ zip: adding: inert_lambda_calculator.py
│ zip: adding: inert_lambda_controller.py
│ zip: adding: tomorrow.py
│ Traceback (most recent call last):
│   File ".terraform/modules/lambda_inert_calculator.lambda/package.py", line 1389, in <module>
│     main()
│   File ".terraform/modules/lambda_inert_calculator.lambda/package.py", line 1385, in main
│     exit(args.command(args))
│   File ".terraform/modules/lambda_inert_calculator.lambda/package.py", line 1277, in build_command
│     bpm.execute(build_plan, zs, query)
│   File ".terraform/modules/lambda_inert_calculator.lambda/package.py", line 318, in __exit__
│     self.close()
│   File ".terraform/modules/lambda_inert_calculator.lambda/package.py", line 308, in close
│     os.replace(self._tmp_filename, self.filename)
│ FileNotFoundError: [Errno 2] No such file or directory:
│ 'builds/c458b42518ccdd1280b4301c65c5cb9ad4204df92543efadf0a7e8108461976e.zip.tmp'
│ ->
│ 'builds/c458b42518ccdd1280b4301c65c5cb9ad4204df92543efadf0a7e8108461976e.zip'

Additional context

We use Terragrunt as a wrapper to Terraform.

[glenthomas]

Sorry to leave a +1 comment, but I am having a similar issue and may be able to add some more detail. In my case I am using the lambda module to deploy a NodeJS function and using the source_path input with 2 directories. My Terraform is running in a GitHub Actions workflow.

First failure:

Error: Provider produced inconsistent final plan

When expanding the plan for
module.lambda_function.aws_lambda_function.this[0] to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .source_code_hash: was
cty.StringVal("W5Q/9KTc830rhw7wy9udI1QYKgUg9sNe6uNfptZpX9M="), but now
cty.StringVal("r3Mho0KL/ucjOnWC6gP3K92xXMU8SKjvAYWz7a1IoS0=").

Second failure:

Error: unable to load "builds/16deeb725b0d8d8e363cc4e39e5afb31b3ef8a0643e3f6acbc3c353b1d363376.zip": open builds/16deeb725b0d8d8e363cc4e39e5afb31b3ef8a0643e3f6acbc3c353b1d363376.zip: no such file or directory

  with module.lambda_function.aws_lambda_function.this[0],
  on /tmp/terraform-data-dir/modules/lambda_function/main.tf line 22, in resource "aws_lambda_function" "this":
  22: resource "aws_lambda_function" "this" {

It completed successfully on the 3rd run.

Terraform 1.2.8 AWS Provider 4.29.0 terraform-aws-lambda module 4.0.1

I have another issue that I think is unrelated so will post separately.

[antonbabenko]

@cahnk I think the problem you have is related to the fact that file .terragrunt-source-manifest is not being ignored during the creation of zip-archive. Terragrunt updates it or changes the timestamp and it produces another hash during apply phase. Could you please try to update patterns (like here) to make sure that it includes only the necessary files in the package?

I use this module with Terragrunt, but my ./src files are outside, and there are no .terragrunt-source-manifest or other hidden files.

@glenthomas Could you provide more code snippets (especially the module's argument source_path and include more debug output (e.g. export TF_LAMBDA_PACKAGE_LOG_LEVEL=DEBUG3 && terraform apply )? Currently, there is not enough information for me to understand or reproduce this issue locally.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[lorengordon]

I was finally able to avoid this problem using this setup:

  recreate_missing_package = false
  ignore_source_code_hash  = true

  source_path = [
    {
      path             = "${path.module}/{{path/to/src}}"
      pip_requirements = true
      patterns         = ["!\\.terragrunt-source-manifest"]
    }
  ]

[antonbabenko]

@lorengordon This is the right pattern for Terragrunt users. Thank you for sharing it here!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.