Error "Provider produced inconsistent final plan" for Application Gateway TLS certs

[devbeard]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.6

AzureRM Provider Version

3.21.1

Affected Resource(s)/Data Source(s)

azurerm_application_gateway

Terraform Configuration Files

resource "azurerm_application_gateway" "appgw" {
[...]
  dynamic "ssl_certificate" {
    for_each = var.applications
    content {
      name                = ssl_certificate.value["cert_name"]
      key_vault_secret_id = data.external.target_kv_tls_cert_lookup[ssl_certificate.key].result.sid
    }
  }

  dynamic "http_listener" {
    for_each = var.applications
    iterator = x
    content {
      name                           = format("listener_%s", x.value["frontend_host_name"])
      protocol                       = "Https"
      require_sni                    = true
      host_name                      = x.value["frontend_host_name"]
      ssl_certificate_name           = x.value["cert_name"]
      frontend_ip_configuration_name = local.frontend_ip_configuration_name
      frontend_port_name             = local.frontend_port_name
    }
  }
[...]
}

### Debug Output/Panic Output

```shell
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for azurerm_application_gateway.appgw to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for .ssl_certificate: planned set element
│ cty.ObjectVal(map[string]cty.Value{"data":cty.StringVal(""), "id":cty.UnknownVal(cty.String), "key_vault_secret_id":cty.UnknownVal(cty.String), "name":cty.StringVal("tls-redacted-domain-com"), "password":cty.StringVal(""),
│ "public_cert_data":cty.UnknownVal(cty.String)}) does not correlate with any element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behaviour

TLS certificates rotated without state issues in provider

Actual Behaviour

After planning and applying 2-3 times, everything works as expected, but always fails on first try

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

[ms-zhenhua]

Hi @devbeard, thank you for reaching out. From content of the error, it seems that the ssl_certificate of tls-redacted-domain-com was changed during the apply. But since the definition of var.applications does not exist in the configuration snippet, I cannot be sure what caused the changes during the apply. Could you kindly help provide a runnable configuration for this issue (you could remove the sensitive information)? Then we can do more investigation on it. Thanks.

[mdimovskihtec]

Hi,

I have the same error.

Terraform Version 1.4.6

AzureRM Provider Version 3.63.0

This is my configuration. The issue appeared after adding cert2. I tried removing the newly added code, also tried upgrading/downgrading both terraform and provider versions, but nothing helped.

module "application_gateway" {
  source                         = "../../modules/application_gateway"
  name                           = local.dev_apgw_name
  location                       = var.location
  location_short                 = var.location_short
  rg_name                        = azurerm_resource_group.rg.name
  tags                           = local.tags
  ...
  ssl_certificates = [
    {
      name                = data.azurerm_key_vault_certificate.cert1.name
      key_vault_secret_id = data.azurerm_key_vault_certificate.cert1.secret_id
    },
    {
      name                = data.azurerm_key_vault_certificate.cert2.name
      key_vault_secret_id = data.azurerm_key_vault_certificate.cert2.secret_id
    }
  ]

----------------------------
resource "azurerm_application_gateway" "app_gw" {
  location            = var.location
  name                = var.name
  resource_group_name = var.rg_name
  
  dynamic "ssl_certificate" {
    for_each = var.ssl_certificates

    content {
      name                = ssl_certificate.value.name
      key_vault_secret_id = ssl_certificate.value.key_vault_secret_id
    }
  }
  ...
}

[mdimovskihtec]

I solved this issue by simply going to the Listeners > Listeners TLS Certificates Tab on Azure Portal, editing the certificate in question and re-selecting the certificate from the key vault. Re-ran terraform and it passed successfully.

[rcskosir]

Thanks for taking the time to submit this issue. @mdimovskihtec had the same error and has listed a solution above. As such, I am going to mark this issue as closed. If this does not solve the issue for you, please reach out.

[barrypitman]

@mdimovskihtec's solution isn't working for me, as I have ~50 certificates, and I can't update all of them in the portal every time I need to make a change to the application gateway. I have been resorting to deleting it and re-creating it, but the issue always resurfaces. I have tried unsuccessfully (so far) to reproduce the issue in a sample project.

[barrypitman]

I also tried to remove the azurerm_application_gateway from the state and import it, and run into the same problem:

 Error: Provider produced inconsistent final plan
│
│ When expanding the plan for
│ module.app_gateway["primary"].azurerm_application_gateway.app_gateway to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
│ .ssl_certificate: planned set element
│ cty.ObjectVal(map[string]cty.Value{"data":cty.StringVal(""),
│ "id":cty.UnknownVal(cty.String),
│ "key_vault_secret_id":cty.UnknownVal(cty.String),
│ "name":cty.StringVal("xxx"),
│ "password":cty.StringVal(""),
│ "public_cert_data":cty.UnknownVal(cty.String)}) does not correlate with any
│ element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[barrypitman]

It seems to be related to the use of versionless_secret_id:

  ssl_certificate {
    name = "xx"
    key_vault_secret_id = data.azurerm_key_vault_certificate.pfx_secret.versionless_secret_id
  }

When I use data.azurerm_key_vault_certificate.pfx_secret.secret_id instead of data.azurerm_key_vault_certificate.pfx_secret.versionless_secret_id, then it seems to work. However, the recommended practice seems to be to refer to versionless_secret_id so that you don't pin your application gateway to a specific certificate

[barrypitman]

Sorry for spamming this issue, but I realised that when referring to the versionless_secret_id, its not necessary to fetch it using data.azurerm_key_vault_certificate, you can just say:

key_vault_secret_id = "${var.key_vault.vault_uri}secrets/${certificate_name.key}"

Seems to have resolved the issue for now

[samrobillard]

Can we continue the investigation in a fix in the provider? None of the solutions here are acceptable. I shouldn't need to do manually edit the application gateway or do work arounds to bypass an azurerm resource output.

[tracetechnical]

The listed workaround is NOT acceptable. The provider does not work as intended.

Please re-open this issue @rcskosir

[rcskosir]

Thanks for reaching out, I can reopen this issue.

[jgresc]

Same issue here. Not sure if others have noticed that as well, but we have several application gateways. The ones with only a few listener, the issue does not occur. With the ones with more (+10), it fails. After re-apply, the issue is gone.

Also as a side note, whenever I add a new listener or make a change in the existing one, all other listeners/certificates/redirect_configuration/request_routing_rule...(essentially all dynamic blocks) get's planned and shown as a diff again, where I am only interessted in the one block I changed. Is there a way to prevent this? Edit: That's prob related to this issue: https://github.com/hashicorp/terraform-provider-azurerm/issues/6896

[rmajid2]

Any update ?

[jorgtho]

What is the status here? I have the same problem