Logic error prevents code updates when `recreate_missing_package = false` in CI/CD scenario. Gives misleading "Provider produced inconsistent final plan" messages on "source_hash" + other weird errors.

[bellis-ai]

TLDR

https://github.com/terraform-aws-modules/terraform-aws-lambda/blob/9d164781174c441c724f86af5486db8fb368282c/main.tf#L50 Produces a logic error that prevents the code from being applied when ran in a pipeline due to files popping in and out of existence. If source hash was simply not used when recreate_missing_package (or the env variable) is false, this problem would probably go away.

Set ignore_source_code_hash = true to bypass this error.

Versions

Module v3.3.1, but pretty much all of them will suffer the same issue Terraform v1.2.1 but pretty much all of them will suffer the same issue

Reproduction Code

module "lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "3.3.1"
  runtime           = "python3.9"
  handler           = "handler.handler"
  function_name     = "my-lambda"
  recreate_missing_package = false
  source_path = [{
    path             = "${path.cwd}/../../my-code"
    pip_tmp_dir      = "${path.cwd}/../"
    pip_requirements = "${path.cwd}/../../my-code/requirements.txt"
  }]
}

Description

This is a longstanding issue that somehow never has been solved somehow (?)

https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/309 https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/259 https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/176 https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/158 https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/144

See this line right here: https://github.com/terraform-aws-modules/terraform-aws-lambda/blob/9d164781174c441c724f86af5486db8fb368282c/main.tf#L50

What's important is that it will only compute the source has if fileexists(local.filename)) and !local.was_missing

Now suppose you're a user of the module deploying this in a pipeline:

• the package won't exist when starting the pipeline
• recreate_missing_package is true
• As a result, was_missing will be false
• However, during the plan step, fileexists(local.filename)) is false since the package doesn't exist.

But during the actual apply, this happens:

• the package is created (before the lambda function is updated)
• the file now exists
• when updating the lambda function, fileexists(local.filename)) and !local.was_missing is true
• Terraform tries grabbing the source hash
• This source hash differs from the previous computed source hash in the plan step (null) and so you receive the following error:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ module.lambda.aws_lambda_function.this[0] to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .source_code_hash: was
│ cty.StringVal("Yo9loSo0h2n7iw+Xu1akz3+lfjr+yVOxSsbatUwP83o="), but now
│ cty.StringVal("t6WnGgoHqpcKihYT6JqBdhDFR/lFdhhnazwvm/MeJGg=").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

(This triggers terraform to back out of trying to update that resource. It never initiates the update)

Suppose you're a frustrated user and you relaunch your pipeline. Doing so, your plan will look like this:

Terraform will perform the following actions:
  # module.lambda.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
      ~ filename                       = "builds/8591bd7f01c79b59a267572ad3a46a599b02bc32a428d4e653cec8c558139d3a.zip" -> "builds/8b65b549a60e6ad8252edca1ee7e51450f7daeddf1a615d1c02ac782af2134db.zip"
        id                             = "my_function"
      ~ last_modified                  = "2022-07-15T20:44:39.708+0000" -> (known after apply)
        tags                           = {}
        # (19 unchanged attributes hidden)
        # (4 unchanged blocks hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

(Notice that only the lambda function will be updated, and the packaging components will not be recreated -- i.e. there will be no deployment package)

Alas, when you apply, it will error upon no deployment package being there:

│ Error: unable to load "builds/8b65b549a60e6ad8252edca1ee7e51450f7daeddf1a615d1c02ac782af2134db.zip": open builds/8b65b549a60e6ad8252edca1ee7e51450f7daeddf1a615d1c02ac782af2134db.zip: no such file or directory
│ 
│   with module.lambda.aws_lambda_function.this[0],
│   on .terraform/my_lambda.lambda/main.tf line 22, in resource "aws_lambda_function" "this":
│   22: resource "aws_lambda_function" "this" {
│ 

But here's where things get funny, if you restart the pipeline and apply a 3rd time, it will appear as if everything worked...

No changes. Your infrastructure matches the configuration.

This is because the terraform aws provider's aws_lambda_function is partially broken, and failed updates to the filename will still update the filename property regardless if the update succeeded

Temporary Solution

Set ignore_source_code_hash = true. As the filename changes upon each new deployment anyways, source code hash does nothing extra anyways. One could also possibly look at using S3 object versioning instead.

[purduemike]

Your temporary solution works for me. Thanks!

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[rodrigocarvalho-tw]

Any updates on this issue?

[antonbabenko]

Please provide a 100% reproducible code snippet and outputs after each run (if possible).

Sometimes I have a hard time understanding this issue completely and I have not hit it myself for a rather long time.

[alizdavoodi]

We also faced this issue in gitlab-ci pipeline. The first problem we encountered was related to the null_resource.archive[0] must be replaced which the fix was provided in FAQ section (recreate_missing_package = false). When we apply recreate_missing_package = false configuration, then we got Provider produced inconsistent final plan" messages on "source_hash"

[antonbabenko]

I have just studied all issues where Provider produced inconsistent final plan was mentioned in this repository and... I was not able to reproduce this problem using the latest version of the module... again :(

I use Terraform 1.2.9 and AWS Provider 4.30.0.

The best suggestion I have come up with while reading and thinking about this problem one more time is to make sure to exclude all irrelevant/system/hidden files and include only the required files into the package (using patterns). I believe that in some cases, .terraform, .terragrunt-cache, and other folders can lead to such issues.

A second suggestion/workaround is to split packaging and deployment as suggested in this comment.

Having 100% reproducible cases would help me to tackle this issue for real. Please provide it to me!

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[lorengordon]

I've also been running into this problem. I believe it is again a problem related to plan vs apply consistency. When first applied in a pipeline, there is no state yet and source_code_hash is created for the first time. No problem. Then in a later run, in a pipeline where the package is being created, the logic around source_code_hash sets it to null in the plan because the package does not yet exist.

Then at apply time, the source_code_hash value is calculated again, but now the package exists, so the value changes. This was fine in the first apply, without any state, but now there is state and terraform blows up.

Since the package gets updated when the filename changes, and the filename is based on the contents of source_path, I don't think I need source_code_hash at all, so I'm just using ignore_source_code_hash = true with this lambda module.

[lorengordon]

I believe https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/345 is actually the same problem.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[tms0]

We also faced this issue in gitlab-ci pipeline. The first problem we encountered was related to the null_resource.archive[0] must be replaced which the fix was provided in FAQ section (recreate_missing_package = false). When we apply recreate_missing_package = false configuration, then we got Provider produced inconsistent final plan" messages on "source_hash"

Exactly the same problem here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

This issue was automatically closed because of stale in 10 days

[dtykalowski-tsh]

Still an issue

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.