GCP SA key creation with "new value: Root resource was present, but now absent"

[mr-tko]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

terraform --version
Terraform v1.7.3 
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.17.0
+ provider registry.terraform.io/hashicorp/time v0.10.0

Affected Resource(s)

google_service_account_key

Terraform Configuration

resource "time_rotating" "mykey_rotation" {
  rotation_minutes = 1
}

resource "google_service_account" "myaccount" {
  count = 15
  account_id   = "test-sa-key-gen-issue-${count.index}"
  display_name = "Testing SA key issue [${count.index}]"
  project = var.project_id
}

resource "google_service_account_key" "mykey" {
  count = 15
  service_account_id = google_service_account.myaccount[count.index].name

  keepers = {
    rotation_time = time_rotating.mykey_rotation.rotation_rfc3339
  }
}

#!/bin/bash

i=0
while [ $i -lt 30 ]
do
   echo $i
   i=`expr $i + 1`
   terraform apply -auto-approve
   sleep 90
done

Debug Output

This issue occurs randomly, so I won't be pasting the code here. Just run the script and it will output error - for me it did at the first run:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to google_service_account_key.mykey[5], provider
│ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present,
│ but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

SA key rotated without and error

Actual Behavior

Randomly an error occurs

provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.

Steps to reproduce

1. sh ./test_script.sh > multiple_sa_key_issue.log

Important Factoids

Error occurs randomly, possibly when you make a lot of operations on the SA or SA keys.

References

Definitely can relate to this issue: https://github.com/hashicorp/terraform-provider-google/issues/13364

b/327622972

[ggtisc]

It looks similar to the issue https://github.com/hashicorp/terraform-provider-google/issues/13364 so maybe you can try adding more time between the google_service_account and google_service_account_key or try with a google_cloud_scheduler_job https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job

[evanyounggenesys]

Is the delay to account for eventual consistency on the Google Cloud Platform side? Our team has encountered this a couple of times but I noticed the resourceGoogleServiceAccountKeyCreate uses a tail call to resourceGoogleServiceAccountKeyRead. It seems like that tail call read could be the issue with the eventually consistency which ends up removing the key from our state file.

I think a delay in the provider would be needed in that case as adding a delay as suggested would be too late and not help. Perhaps better yet would to be remove the tail call resourceGoogleServiceAccountKeyRead entirely and let any delay or validation happen for whatever reads the value later.

[evanyounggenesys]

@edwardmedia Can you take a look at this issue considering your delay suggestions from #13364? It seems like something that needs to be addressed by the provider if my rudimentary understanding of Terraform and Go is correct.

[mcallaghan-geotab]

we are also experiencing this intermittently

when it fails, it fails in similar way as OP

Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.foo.module.workload_identity["iat"].google_service_account.cluster_service_account[0],
│ provider "provider[\"[registry.terraform.io/hashicorp/google\](http://registry.terraform.io/hashicorp/google%5C)"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

— but then we retry— and in it says the resource already exists

Error: Error creating service account: googleapi: Error 409: Service account foo-iat already exists within project projects/geotab-foobar.
│ Details:
│ [
│   {
│     "@type": "[type.googleapis.com/google.rpc.ResourceInfo](http://type.googleapis.com/google.rpc.ResourceInfo)",
│     "resourceName": "projects/foobar/serviceAccounts/[email protected]"
│   }
│ ]
│ , alreadyExists
│ 
│   with module.foo.module.workload_identity["iat"].google_service_account.cluster_service_account[0],
│   on .terraform/modules/foo.workload_identity/modules/workload-identity/[main.tf](http://main.tf/) line 42, in resource "google_service_account" "cluster_service_account":
│   42: resource "google_service_account" "cluster_service_account" {
│

[mr-tko]

I've got new facts: it's happening for sure, when you delete old and create a new key for the service account. As a proof, I can only attach a small portion of obfuscated logs (as it is happening constantly on a production env). First log says that key was present in the state file:

module.XYZ.google_service_account_key.sa_key: Refreshing state... [id=projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE]

This log presents the plan portion and later on the error message:

module.XYZ.google_service_account_key.sa_key must be replaced
+/- resource "google_service_account_key" "sa_key" {
      ~ id                 = "projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE" -> (known after apply)
      ~ keepers            = {
          - "rotation_time" = "2024-05-XXTXX:XX:XXZ"
        } -> (known after apply) # forces replacement
      ~ name               = "projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE" -> (known after apply)
...
.
.
.
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.XYZ.google_service_account_key.sa_key,
│ provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.