Error: timed out while waiting for user to be inserted

[xens]

Hi there,

I'm hitting a random but very frequent issue when creating Google Workspace users. The user creation process takes usually a lot of time (5 minutes per user) and most of the time it times out.

Terraform Version

$ terraform -v
Terraform v1.3.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.42.0
+ provider registry.terraform.io/hashicorp/googleworkspace v0.7.0
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.0

Affected Resource(s)

• googleworkspace_user

Terraform Configuration Files

resource "random_password" "password" {
  length = 30
  special = true
}

resource "googleworkspace_user" "users" {

  primary_email = "[email protected]"
  password      = md5(random_password.password.result)
  hash_function = "MD5"

  org_unit_path = "/myOrg"

  name {
    family_name = "userFamiliyName"
    given_name  = "userGivenName"
  }

  organizations {
    department = "my Dept"
    primary    = true
    type       = "work"
  }
  recovery_email = "[email protected]"

    timeouts {
    create = "15m"
  }

Expected Behavior

User should be properly created.

Actual Behavior

90% of the time the user creation fails with the following error

googleworkspace_user.users["[email protected]"]: Still creating... [6m20s elapsed]
googleworkspace_user.users["[email protected]"]: Still creating... [6m30s elapsed]
googleworkspace_user.users["[email protected]"]: Still creating... [6m40s elapsed]
googleworkspace_user.users["[email protected]"]: Still creating... [6m50s elapsed]
│ Error: timed out while waiting for user to be inserted
│ 
│   with googleworkspace_user.users["[email protected]"],
│   on users.tf line 14, in resource "googleworkspace_user" "users":
│   14: resource "googleworkspace_user" "users" {

Steps to Reproduce

1. terraform apply

[xens]

I'll need to sanitize a bit my TF_LOG before uploading it but I've been able to dig a bit more into this issue already and looks like its linked to the resource consistencyCheck not being met.

According to this file, the number of consistent responses from the Google Workspace API must equal 4.

// The number of consistent responses we want before we consider the resource consistent
const numConsistent = 4

And it uses the response etag value to make these consistency checks.

By grep'ing the responses etags in TF_LOG=trace we can see that its value is often flipping between two different values and thus not meeting the consistencyCheck condition.

examples when it errors:

 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/-uCe4x3GsEUVT6jHRPQf0qnDKKE\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",

And when it succeeds we can see that the condition is met

 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/q6_2E4VfX6skeuoFnf4_SITEGww\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",

[xens]

Wondering if it could be linked to https://github.com/hashicorp/terraform-provider-googleworkspace/issues/263

[xens]

I did some more tests today and it looks like that Google APIs are returning more consistent etags this morning and thus the creation process succeeds more frequently

googleworkspace_user.users["[email protected]"]: Creation complete after 1m48s [id=103959683524035821588]
googleworkspace_user.users["[email protected]"]: Creation complete after 1m48s [id=114270762350499200959]
googleworkspace_user.users["[email protected]"]: Creation complete after 1m48s [id=106563825273091153419]
googleworkspace_user.users["[email protected]"]: Creation complete after 1m58s [id=101969177768374119408]

Apply complete! Resources: 8 added, 0 changed, 1 destroyed.

Outputs:

user_email = [
  "[email protected]",
  "[email protected]",
  "[email protected]",
  "[email protected]",
]

and of course I can still correlate the creation successes with a series of consistent etags corresponding to the resource.

grep "\"etag" out.log | grep "qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU" 
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",

So at this point I'm wondering if 4 consecutive etags are not too conservative for this kind of resource ? How to make this process more robust?

[xens]

Ok problem is back today with the inconsistent etags :sweat_smile:, for now I've edited the consistencyCheck and compiled + deployed manually the provider following this document.

- const numConsistent = 4
+ const numConsistent = 2

and it's working like a charm.

[soloradish]

We encounter this problem as well. We just rm the user from the state and import it back.

[SamuZad]

This is an issue relating to the google APIs eventual consistency, and how the provider handles the checks for said eventual consistency

I significantly improved this in my fork: https://registry.terraform.io/providers/SamuZad/googleworkspace/latest/docs

The specific commit can be found here: https://github.com/SamuZad/terraform-provider-googleworkspace/commit/86270a6628f23caaf7455909d686d8e58912d074

Since implementing this, timing out during this operation has gone away completely, with no other problems cropping up 🙂