terraform-provider-aws Lambda occasionally doesn’t persist to the statefile

Terraform 0.12.30 Provider 3.39.0 We create a role, a policy, a policy attachment and a lambda from a json list.

this means that we are using foreach with a depends clause to keep the graph connected correctly. We do see the role created, then the policy created and then the attachment correctly, then finally the lambda. The graph is in order and everything completes before the next resource is created.

On occasion, lambda can’t create the eni and fails with the appropriate error message, but sometimes there is a lambda in Aws but it is not in the statefile. This happens maybe 1 in 25 deployments, we do red/black so I’m starting to see this a lot now that we deploy 30 lambdas in a single pass.

I will paste the logs in here soon, I work at a bank so I’m having to do this on my cell phone for now. BUT, like I say, the graph is correct, just when the lambda can’t create the eni for a vpc and it fails in a way where the lambda is not persisted to the statefile and we have to clean up the lambda by hand or import it into the statefile. Any ideas?

The error message that triggers this failure is on ../modules/app-region/lambda.tf line 12, in resource "aws_lambda_function" "integration":

12: resource aws_lambda_function integration {

Error: error waiting for Lambda Function (ls-black-finxact-position-detail-service) creation: InsufficientRolePermissions: The function's execution role doesn't have permission to perform this operation.

May 14 '21 15:05 plainsane

We're facing a similar issue with the exact same error message (This only happens when we're recreating the lambda, updating the lambda happens without any error messages):

Error: error waiting for Lambda Function (last_attribution) creation: InsufficientRolePermissions: The function's execution role doesn't have permission to perform this operation.

This issue happens occasionally even with the following providers:

Installing hashicorp/aws v4.0.0...
Installed hashicorp/aws v4.0.0 (signed by HashiCorp)

We will try to upgrade to Terraform 1.1.6 from 1.0.1 to see if that makes a difference. (Running on AWS codebuild with Alpine linux) Although it would be great to know why this issue is happening.

Feb 20 '22 14:02 csabca83

~~When the error happens, removing the tainted lambda from the state and importing it from aws fixes the state.~~

It doesn't seem to work, the state of the lambda on AWS seems to be incorrect as the warning message is also showing on the console. Only solution when the error happens seems to manually delete the lambda and re-create it again.

Versions: Terraform v1.1.2 and aws v3.68.0