terraform-provider-google icon indicating copy to clipboard operation
terraform-provider-google copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

Service Account │ Error: Provider produced inconsistent result after apply

Open bgalkows opened this issue 1 year ago • 9 comments

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to a user, that user is claiming responsibility for the issue.
  • Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.5.7 on Ubuntu

  • provider registry.terraform.io/hashicorp/google v4.8.3
  • provider registry.terraform.io/hashicorp/google-beta v4.8.3

Affected Resource(s)

google_service_account

Terraform Configuration

terraform {
  backend "gcs" {
    bucket = "..."
    prefix = "..."
  }

  required_version = "~> 1.5.0"
  required_providers {
    google = {
      version = "~> 4.65, < 4.84"
      source  = "hashicorp/google"
    }
    google-beta = {
      version = "~> 4.65, < 4.84"
      source  = "hashicorp/google-beta"
    }
  }
}

Debug Output

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to │ module.[...].google_service_account.service_accounts[...], │ provider "provider["registry.terraform.io/hashicorp/google"]" produced an │ unexpected new value: Root resource was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker.

Expected Behavior

A service account should have been actuated normally.

Actual Behavior

After an apply error, the service account was created successfully in GCP but never acknowledged in Terraform state. An import block was needed to resolve the situation.

Steps to reproduce

  1. terraform apply, creating a google_service_account resource

Important Factoids

No response

References

No response

b/341145887

bgalkows avatar May 10 '24 16:05 bgalkows

This issue occurs several times a day also with version 5.x. The problem might be that the providers is not using long enough timeouts for the 404 retries to deal with eventual consistency while creating service accounts. Adding a sleep after resource creation won't help as the provider fails during creation .

Terraform 1.8.3
provider "registry.terraform.io/hashicorp/google" {
  version     = "5.25.0"

pebo avatar May 16 '24 08:05 pebo

To provide some further information on the issue

Terraform 1.8.1
Google provider 5.25.0

Some debug logs from when this occurs

Debug logs 
### Initial Post to create SA

2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: POST /v1/projects/a-project/serviceAccounts?alt=json&prettyPrint=false HTTP/1.1
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: iam.googleapis.com
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.8.1 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Length: 181
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "accountId": "sa-name",
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "serviceAccount": {
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "description": "TERRAFORMED: access needed for service/service-name(seed-string)",
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "displayName": "service/service-name"
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  }
2024-05-15T10:23:07.088+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }

### POST response

2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 200 OK
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:07 GMT
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "name": "projects/a-project/serviceAccounts/[email protected]",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "projectId": "a-project",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "uniqueId": "00000000000000",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "email": "[email protected]",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "displayName": "service/service-name",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "etag": "MDEwMjE5MjA=",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "description": "TERRAFORMED: access needed for service/service-name(seed-string)",
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "oauth2ClientId": "00000000000000"
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }

### Verify SA exists with GET

2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /v1/projects/a-project/serviceAccounts/[email protected]?alt=json&prettyPrint=false HTTP/1.1
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: iam.googleapis.com
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.8.1 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:07.813+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5

### GET response 404
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 404 Not Found
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "error": {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "code": 404,
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "errors": [
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "domain": "global",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "reason": "notFound"
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     ],
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "status": "NOT_FOUND"
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found

### GET response Retry? 404
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "error": {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "code": 404,
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "errors": [
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       {
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "domain": "global",
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "reason": "notFound"
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     ],
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "status": "NOT_FOUND"
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [DEBUG] Retry Transport: Returning after 1 attempts
2024-05-15T10:23:08.063+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [DEBUG] Dismissed an error as retryable. Retry 404s for service account creation - googleapi: Error 404: Service account projects/a-project/serviceAccounts/[email protected] does not exist., notFound

### Verify SA exists with GET

2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /v1/projects/a-project/serviceAccounts/[email protected]?alt=json&prettyPrint=false HTTP/1.1
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: iam.googleapis.com
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.8.1 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.565+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5

### GET response 200

2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 200 OK
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "name": "projects/a-project/serviceAccounts/[email protected]",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "projectId": "a-project",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "uniqueId": "00000000000000",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "email": "[email protected]",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "displayName": "service/service-name",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "etag": "MDEwMjE5MjA=",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "description": "TERRAFORMED: access needed for service/service-name(seed-string)",
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "oauth2ClientId": "00000000000000"
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }

### Verify SA exists with GET

2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /v1/projects/a-project/serviceAccounts/[email protected]?alt=json&prettyPrint=false HTTP/1.1
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: iam.googleapis.com
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.8.1 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.708+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5

### GET response 200

2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 200 OK
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "name": "projects/a-project/serviceAccounts/[email protected]",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "projectId": "a-project",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "uniqueId": "00000000000000",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "email": "[email protected]",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "displayName": "service/service-name",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "etag": "MDEwMjE5MjA=",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "description": "TERRAFORMED: access needed for service/service-name(seed-string)",
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:  "oauth2ClientId": "00000000000000"
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }

### Verify SA exists with GET

2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /v1/projects/a-project/serviceAccounts/[email protected]?alt=json&prettyPrint=false HTTP/1.1
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: iam.googleapis.com
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.8.1 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.842+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5

### GET response 404

2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 404 Not Found
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "error": {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "code": 404,
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "errors": [
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "domain": "global",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "reason": "notFound"
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     ],
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "status": "NOT_FOUND"
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5

### GET response Retry? 404

2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 404 Not Found
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "error": {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "code": 404,
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "errors": [
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "domain": "global",
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "reason": "notFound"
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     ],
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "status": "NOT_FOUND"
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: -----------------------------------------------------
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: private
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 15 May 2024 08:23:08 GMT
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: ESF
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Referer
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Content-Type-Options: nosniff
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Frame-Options: SAMEORIGIN
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Xss-Protection: 0
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   "error": {
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "code": 404,
2024-05-15T10:23:08.982+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "errors": [
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       {
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "message": "Service account projects/a-project/serviceAccounts/[email protected] does not exist.",
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "domain": "global",
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:         "reason": "notFound"
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:       }
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     ],
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:     "status": "NOT_FOUND"
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5:   }
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [DEBUG] Retry Transport: Returning after 1 attempts
2024-05-15T10:23:08.983+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/05/15 10:23:08 [WARN] Removing Service Account "projects/a-project/serviceAccounts/[email protected]" because it's gone
2024-05-15T10:23:08.985+0200 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-05-15T10:23:08.985+0200 [ERROR] vertex "module.a_service.module.service.module.sa.google_service_account.accounts[0]" error: Provider produced inconsistent result after apply

dahel8 avatar May 16 '24 08:05 dahel8

Unable to reproduce, many users are reporting the same issue so I'm directly forwarding this

ggtisc avatar May 16 '24 21:05 ggtisc

It seems like it is easier to reproduce this problem in the morning (US timezones), like from 9a-12p MDT. It seems like, maybe the provider tries to create the resource, then lookup some additional information or something, the way the error message is written. It doesn't exist for the info query, so it fails. It would be nice if it would create it as tainted at the very least in the state file so that a retry of the pipeline would work.

We are using provider version v5.27.0 and TF 1.8.3.

TJM avatar May 17 '24 20:05 TJM

It seems like it is easier to reproduce this problem in the morning (US timezones), like from 9a-12p MDT.

Confirmed we've been observing similar problems for the past months at least 5 times. Coincidentally, it happened to be within this time range for the most recent instance.

TPG v5.30.0
Terraform v1.5.6

provider "google" {
  request_timeout = "120s"
}

resource "google_service_account" "<>" {
  // ...
}

michaellzc avatar Jun 13 '24 15:06 michaellzc

https://github.com/hashicorp/terraform-provider-google/blob/98cf8d948627a89dafd621ff6ce457e4165c4374/google/services/resourcemanager/resource_google_service_account.go#L300-L303

https://github.com/hashicorp/terraform-provider-google-beta/blob/5bf43efa17854299b0335e8175750e83b073694c/google-beta/services/resourcemanager/resource_google_service_account.go#L163-L166

google-beta provider v5.38.0 waits 10 sec prior to reading the created service account (vs 5 sec at the google provider) and solve the inconsistency issue (most of the time).

the google IAM api is eventually consistent hence reading the service account metadata right after writing a new service account might not result with the latest mutation of the resource.

https://google.aip.dev/121#strong-consistency

https://cloud.google.com/iam/docs/overview#consistency

this behavior violates the Terraform Resource Instance Change Lifecycle and that's pretty much the only solution currently for applying changes to eventually consistent API's. https://github.com/hashicorp/terraform/blob/main/docs/resource-instance-change-lifecycle.md

upgrade ur version to the latest one and use google-beta provider to provision service account's and bind them to IAM policies, this way the provider would wait 10 sec before reading its state and writing it to state file.

oavner avatar Jul 18 '24 08:07 oavner

This bug has been "dodged" in the release v5.32.0 of the "regular" provider, with an unexpected time.Sleep(10 * time.Second) :sweat_smile:.

iam: added a 10 second sleep when creating a 'google_service_account' resource to reduce eventual consistency errors(https://github.com/hashicorp/terraform-provider-google/pull/18261)

"Fixed" by https://github.com/hashicorp/terraform-provider-google/pull/18261

Ameausoone avatar Aug 28 '24 12:08 Ameausoone

Increasing the timeout is not a real fix. Tomorrow you'll have to increase it again.

https://www.commitstrip.com/en/2017/05/22/a-story-about-callbacks/

Exagone313 avatar Aug 28 '24 12:08 Exagone313

To make things more clear, we are hit by this issue on other resources too, such as a google_sourcerepo_repository, where no time.Sleep is done in the implementation. Most of the API are eventually consistent. But it is difficult to deal with this with Terraform.

Please do not close this issue until a more long term solution is implemented for all resources.

Exagone313 avatar Aug 28 '24 15:08 Exagone313