Api Gateway Rest Api Redeployment Fails Generating inconsistent state file

[h3ct0rjs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

• aws_api_gateway_deployment

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

Using hashicorp/aws "3.57.0"
Terraform version: 1.0.3

resource "aws_api_gateway_deployment" "s_rest_api" {
    rest_api_id = aws_api_gateway_rest_api.s_rest_api.id

    triggers = {
        redeployment = sha1(
            join(",", [
                jsonencode(aws_api_gateway_integration.s_rest_api_title_x_children_get),
                jsonencode(aws_api_gateway_method.s_rest_api_title_x_versions_get),
                jsonencode(data.aws_iam_policy_document.title_api_ip_allow) 
            ])
        )
    }
    lifecycle {
        create_before_destroy = true
    }
}

Debug Output

2021-09-06T18:12:12.579-0500 [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value for aws_api_gateway_method.s_rest_api_title_x_versions_get, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .authorizer_id: was cty.StringVal("7gtrfj"), but now cty.StringVal("")
2021-09-06T18:12:12.586-0500 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.57.0/linux_amd64/terraform-provider-aws_v3.57.0_x5 pid=16991
2021-09-06T18:12:12.586-0500 [DEBUG] provider: plugin exited

Panic Output

Expected Behavior

Update an api gateway rest api resource, base on the previous state. After doing multiple modifications of the redeployment part it seems like Terraform is no able to run the plan properly.

Actual Behavior

Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.title-api.aws_api_gateway_deployment.dam_services_rest_api to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .triggers["redeployment"]: was cty.StringVal("bb7cc004678be1cc91507b0d0824e63fceaaaa42"), but now
│ cty.StringVal("6e533493f10740ba9c06ad2024c78771efdea841").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Steps to Reproduce

create an api gateway resource like in the previous code, after them create multiple triggers and later on update the policy or one of the integrations.

1. terraform apply

Important Factoids

References

• #17341

[h3ct0rjs]

Hey guys,

After playing a lot with this it seems like I identified the issue that is causing me a big trouble, I just changed the previous Terraform snippet of code to the following:

resource "aws_api_gateway_deployment" "s_rest_api" {
    rest_api_id = aws_api_gateway_rest_api.s_rest_api.id

    triggers = {
        redeployment = sha1( jsonencode( [
                aws_api_gateway_integration.s_rest_api_title_x_children_get,
                aws_api_gateway_method.s_rest_api_title_x_versions_get,
               data.aws_iam_policy_document.title_api_ip_allow 
            ])
        )
    }
    lifecycle {
        create_before_destroy = true
    }
}

If I comment out the aws_api_gateway_method.s_rest_api_title_x_versions_get resource the redeployment works like a charm. The problem that I identified if I'm not wrong is the resource aws_api_gateway_deployment is going to create the redeployment, acording to the logs it needs to compute a hash in this case a sha1 hash and later on when Terraform is going to apply that it seems that is something that is causing this hash to be computed again and change. after comment out the method trigger it works, so I don't know why is not working with the method resource, is there any logical Explanation ?

Regards, H

[h3ct0rjs]

Hey @ewbankkit I think that this is a bug not just a question. I have reviewed multiple repos of code over internet and inside different 500 companies and it seems like nobody knows what is the correct order for the dependency part using the api gateway resource, this is a blocker, and people over the internet is just guessing which is the way to specify the explicit dependency order.

Things like this is doing that most of the time people switch to use tools like sam or sls, so please take at look at this, this could also be avoided if we add another example under the examples as a devops I prefer to use Terraform but I also adapt to rapid changes and company decisions

Regards, H

[dvins]

This is definitely a bug; the inconsistency here and inability to consistently apply updates such as to Headers without ultimately having to fall back to manually destroying and then recreating the entire resource is a huge problem and we too are now falling back to other, non -TF approach here.

[Arlington1985]

Hey guys,

After playing a lot with this it seems like I identified the issue that is causing me a big trouble, I just changed the previous Terraform snippet of code to the following:

resource "aws_api_gateway_deployment" "s_rest_api" {
    rest_api_id = aws_api_gateway_rest_api.s_rest_api.id

    triggers = {
        redeployment = sha1( jsonencode( [
                aws_api_gateway_integration.s_rest_api_title_x_children_get,
                aws_api_gateway_method.s_rest_api_title_x_versions_get,
               data.aws_iam_policy_document.title_api_ip_allow 
            ])
        )
    }
    lifecycle {
        create_before_destroy = true
    }
}

If I comment out the aws_api_gateway_method.s_rest_api_title_x_versions_get resource the redeployment works like a charm. The problem that I identified if I'm not wrong is the resource aws_api_gateway_deployment is going to create the redeployment, acording to the logs it needs to compute a hash in this case a sha1 hash and later on when Terraform is going to apply that it seems that is something that is causing this hash to be computed again and change. after comment out the method trigger it works, so I don't know why is not working with the method resource, is there any logical Explanation ?

Regards, H

That's not quite correct. When you trigger deployment with data.aws_iam_policy_document.title_api_ip_allow, and also change policy it might happen that deployment will start before the policy update which is not right. The intention here is to redeploy every time after the policy was updated. Otherwise, that change will not be included in the last deployment. Basically, I am looking for a way how to redeploy exactly in the end.

[Arlington1985]

@ewbankkit do you know a workaround for the situation above I described? I feel like it's definitely a bug

[Arlington1985]

As a workaround, we added a "hacky" way of doing things: second terraform plan and apply stage. if you know better way please recommend

[ahached]

Hello like Arlington1985, we were forced to use a second plan and apply stage. Any chance that this will get solved ? the workaround is horrible :/

[AlecTaggart]

also really would love a fix for this issue ...

[mkandelaars]

I have recently spent some time looking into this same issue and have a workaround in place. I believe the issue is to do with the normalisation of the JSON in the aws_api_gateway_rest_api_policy resource, as it seems that the value of the policy changes on the 2nd terraform apply with only formatting changes.

Similarly to @Arlington1985 and @ahached, performing a 2nd terraform plan / apply isn’t a great solution (although it does work), and that relying on a change in the policy document (e.g. data.aws_iam_policy_document.title_api_ip_allow) in the deployment trigger could also result in inconsistent results as also mentioned, so definitely best to avoid that as a solution.

Indeed it looks to be a bug, but the workaround for me is to perform the JSON normalisation manually by wrapping the policy in a jsonencode(jsondecode()) in the deployment trigger. e.g. doing something like…

resource "aws_api_gateway_deployment" "s_rest_api" {
    rest_api_id = aws_api_gateway_rest_api.s_rest_api.id

    triggers = {
        redeployment = sha1( jsonencode( [
                aws_api_gateway_integration.s_rest_api_title_x_children_get,
                aws_api_gateway_method.s_rest_api_title_x_versions_get,
                jsonencode(jsondecode(aws_api_gateway_rest_api_policy.rest_api_policy.policy)) // <———
            ])
        )
    }
    lifecycle {
        create_before_destroy = true
    }
}

This was first suggested in the other thread https://github.com/hashicorp/terraform-provider-aws/issues/17341#issuecomment-822575936, and they also mention some important caveats to consider, but their reasoning sheds a lot of light on the underlying issue.

[Arlington1985]

@mkandelaars I have already tried this approach it has no effect. I guess the reason is simply policy doesn't apply before deployment and stage change

[mkandelaars]

@Arlington1985 Are you sure your code for the trigger was on the aws_api_gateway_rest_api_policy object (e.g. aws_api_gateway_rest_api_policy.rest_api_policy.policy)? I noticed in your original snippet you had data.aws_iam_policy_document.title_api_ip_allow in the trigger, but that will always result in a race condition with the deployment. If you use the value of the policy in the aws_api_gateway_rest_api_policy resource in the trigger, it will only apply after the resource policy is updated.

[Arlington1985]

@mkandelaars the code running inside a module. api_policy is not null, json policy sent as parameter

resource "aws_api_gateway_rest_api_policy" "rest_api_policy" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  policy      = var.api_policy
}

resource "aws_api_gateway_deployment" "api_gateway_deployment" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  triggers = {
    redeployment = sha1(jsonencode([
      aws_api_gateway_rest_api.rest_api.body,
      var.api_policy == "" ? null :  jsonencode(jsondecode(aws_api_gateway_rest_api_policy.rest_api_policy.policy))
    ]))
  }

[mkandelaars]

@Arlington1985 - I would have expected that to work. If you remove the jsonencode(jsondecode wrapping do the get the inconsistent plan error on apply? If so, then that confirms the Resource Policy will get updated before the Deployment is created since it is using the value of the policy after it’s been updated (which is inconsistent with the expected value and the ultimate cause of this issue) That is the behaviour I see on Terraform 1.5.1 with AWS Provider v5.5.0 and is consistent with the output of terraform graph in terms of the order of things

[neironus]

In my case the root of the problem was aws_api_gateway_rest_api_policy. The error appears If you are using "Resource": ["execute-api:/*"]

# Example without errors
resource "aws_api_gateway_rest_api_policy" "this" {
  count = var.private_gateway ? 1 : 0

  rest_api_id = aws_api_gateway_rest_api.this.id
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : "*",
        "Action" : "execute-api:Invoke",
        "Resource" : "arn:aws:execute-api:${local.aws_region}:${local.aws_id}:${aws_api_gateway_rest_api.this.id}/${var.environment}/*/*",
        "Resource" : "*"
      }
    ]
  })
}