Terraform Google Project Factory Module Inconsistent Result Error "Error: Provider produced inconsistent result after apply"

[oriann-bar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.1.9 on

• provider registry.terraform.io/hashicorp/google v5.27.0
• provider registry.terraform.io/hashicorp/google-beta v5.27.0

Affected Resource(s)

google_service_account.service_accounts["gke-nodes"]

Terraform Configuration

data "google_organization" "org" {
  domain = var.domain
}

module "project" {
  source                     = "terraform-google-modules/project-factory/google"
  version                    = "15.0.0"
  name                       = var.project_name
  random_project_id          = false
  org_id                     = data.google_organization.org.org_id
  folder_id                  = var.folder_id
  billing_account            = var.billing_account
  default_service_account    = var.default_service_account
  lien                       = var.lien
  disable_dependent_services = true
  activate_apis              = var.active_apis
}

resource "google_compute_project_metadata_item" "default" {
  for_each = toset(keys(var.metadata))
  key      = each.key
  value    = var.metadata[each.key]
  project  = module.project.project_id
}

resource "google_service_account" "service_accounts" {
  for_each   = toset(distinct(var.service_accounts))
  project    = module.project.project_id
  account_id = each.key
}

Debug Output

No response

Expected Behavior

No response

Actual Behavior

When utilizing the terraform-google-modules/project-factory/google module (version 15.0.0) with the Google provider (version 5.27.0), the Terraform apply operation intermittently fails with an "inconsistent result" error. This error occurs randomly and persists even after updating the module version (version 15.0.0) and provider versions to 5.27.0.

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to │ google_service_account.service_accounts["gke-nodes"], provider │ "provider["registry.terraform.io/hashicorp/google"]" produced an │ unexpected new value: Root resource was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker. ╵ time=2024-05-06T08:04:13Z level=error msg=Module /workspace/louhi_ws/terragrunt/Environments/Staging/us-central1/n9ybd/project has finished with an error: 1 error occurred: * exit status 1

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

b/339515483

[ture-karlsson]

I get this too since yesterday. It appears seemingly random and not on every apply.

I run Terraform 1.7.1 through TFE with latest version of the provider:

required_providers {
  google = {
    source  = "hashicorp/google"
    version = ">= 5.16, < 6"
  }
}

I suspected it was because of 5.28.0 that was released yesterday but in the report above it's on 5.27.0 as well.

Here's the error:

Error: Provider produced inconsistent result after apply
When applying changes to module.<module_name_omitted>.google_service_account_key.<key_name_omitted>, provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

[shaikatzz]

Got this as well starting today, just want to add that I think it's not related to any specific provider version as we are using 5.19.0 and it wasn't change for a while.

[rileykarson]

This error generally indicates the API is not strongly consistent, a violation of GCP's API guidance for management plane resources like ServiceAccountL https://google.aip.dev/121#strong-consistency. We do poll for the resource after creation to mitigate an eventual consistency issue (https://github.com/hashicorp/terraform-provider-google-beta/blob/c1d0fc2b51234089b79ff8fb529105ee6bd4742f/google-beta/services/resourcemanager/resource_google_service_account.go#L155-L157) in the data plane of IAM APIs- technically not breaking that guidance- is eventually consistent. However, this issue comes from the management plane of the API and should be strongly consistent.

[c2thorn]

The goal with https://github.com/GoogleCloudPlatform/magic-modules/pull/10813 is to add more mitigation in the form of a 10 second sleep. This should capture the higher end of the recent eventual consistency issues.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.