plan hangs forever when Storage bucket deleted outside of TF

[jstange]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Both of 3.90.1 (the provider this code had when I found it) and 5.16.0 (which I upgraded to) exhibit this behavior.

Affected Resource(s)

One or both of google_storage_bucket and/or google_storage_bucket_iam_policy.

The presence of these objects in state appears to be the trigger. If I remove them from configuration, the infinite hang persists. However, after I terraform state rm both the bucket resource and its IAM policy resource, the issue goes away.

Terraform Configuration

resource "google_storage_bucket" "bucket-foo" {
  default_event_based_hold = "false"
  force_destroy            = "false"
  location                 = "US"

  logging {
    log_bucket        = "a-logging-bucket-which-yes-does-exist"
    log_object_prefix = "some_prefix"
  }

  name                        = "bucket-foo"
  project                     = "project-foo"
  requester_pays              = "false"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = "false"
}

resource "google_storage_bucket_iam_policy" "bucket-foo" {
  bucket = "b/bucket-foo"

  policy_data = <<POLICY
{
  "bindings": [
    {
      "members": [
        "projectEditor:parent-project",
        "projectOwner:parent-project"
      ],
      "role": "roles/storage.legacyBucketOwner"
    },
    {
      "members": [
        "projectViewer:parent-project"
      ],
      "role": "roles/storage.legacyBucketReader"
    }
  ]
}
POLICY
}

Debug Output

https://gist.github.com/jstange/c4745b0ca270a310f3face7510fececa

Expected Behavior

Bucket and policy are declared in configuration and exist in state, but missing on cloud provider (someone did an unadvised hand-deletion). terraform plan should note this inconsistency, and either throw an error or present a plan to converge on expected state.

Actual Behavior

A plain terraform plan hangs silently and indefinitely.

With debug logging enabled, it appears to be retrying retrieval of the missing bucket. A message indicates "stopping retries," but then the retries continue.

Steps to reproduce

1. Create a bucket and perhaps a bucket IAM policy in configuration, and apply
2. Manually delete the bucket with the GCP console or CLI tools
3. See if TF_LOG=DEBUG terraform plan loops forever trying to update state of the missing bucket

Important Factoids

This statefile was originally created by version 3.90.1 of the provider, which may mean something. Upgrading to 5.16.0 didn't correct things, however.

References

No response

[hao-nan-li]

What is the duration from the first retry until you manually stop the process? For this resource I do see a retry set if the operation returns a 404. There should be a timeout set for 4 minutes.

[jstange]

I wasn't measuring explicitly, but somewhere in the range of 15-30 minutes on the initial attempt.

[melinath]

This looks like a duplicate of https://github.com/hashicorp/terraform-provider-google/issues/13464

[ggtisc]

Confirmed issue: By following the steps mentioned it enter an infinite loop retrying retrieval of the missing bucket.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.