Error: timed out while waiting for group to be updated

[dwilliams782]

Hi,

We are running the latest provider, 0.6.0, and terraform 0.14.7.

One of our users has been experiencing time outs a lot when doing large applies recently.

Our latest PR is adding approx 60 users to an existing group, and we are hitting the configured timeouts:

Error: timed out while waiting for group to be updated

  on modules/_components/iam/google-group/group.tf line 3, in resource "googleworkspace_group" "group":
   3: resource "googleworkspace_group" "group" {

We have the create and update timeouts configured to 5 minutes, but this is causing applies to fail after a long time. Typically in my experience they either work within a few seconds or they don't.

Frustratingly, we apply via runatlantis so the debug logs are abstracted from us. I'm just working on retrieving these logs, but in the mean time, are there any known issues here? Perhaps we're being rate limited by google?

[dwilliams782]

Ok I have debug logs now. During the plan I can see a few oddities:

https://gist.github.com/dwilliams782/ac1d1d9d285f7a4656282b6084900604

And the apply:

https://gist.github.com/dwilliams782/020584420f5365336a579756a07d5301

You may need more of the logs, I've just taken the last few entries for the apply errors. It looks like the member objects are returning a 200 OK but the provider continues trying to add them anyway?

[dwilliams782]

I have noticed that the amount of members to add is reducing each time, so for now I guess we'll just keep planning and applying until it all goes through.

[jazzlyn]

Still an issue and a critical bug. Any updates on this? Interrupting and reapplying manually is neither a solution nor a workaround.

[megan07]

@dwilliams782 Are the debug logs you provided a subset of the whole run?

This seems to be an eventual consistency issue, and we handle these as best we can. My suggestion is to increase the timeout. While I realize this could lead to long-running processes, it's the best we can do when trying to estimate the average time each resource (in a large sample of resources) will take.

I'd be happy to review more configurations and debug logs that would help me to fix our retry logic on this, but from the current debug log provided, I can't tell how long the process had been retrying and what the timeout was.

[dwilliams782]

Hi Mega, yes that is a subset, the entire log would be huge. This was a while ago now but we tried with timeouts up to 10 minutes.

Perhaps @jazzlyn can provide a more recent debug log as I haven't looked at this for a few months now.

[jazzlyn]

Hi guys, Hi @megan07,

I have exact the same issues as @dwilliams782 and those issues can occur already with 1 resource, so I'm creating: 1 googleworkspace_group 1 googleworkspace_group_settings 1 googleworkspace_group_member and randomly get timeouts up to 5m.

The first resource is already there but provider tries to add it continuously. After some time it succeedes or it timeouts at 5m. I've observed that the resources are created in Googleworkspace but are not proper written to state. If I delete the tainted state, I'm able to import the resource properly.

This should be easy to reproduce, I use this module: https://registry.terraform.io/modules/strg-at/user-group-management/googleworkspace/1.0.0

Best, J.

[xens]

Facing the same issue, is there something I could provide to help?

[red8888]

This has been open for a year is this really still the state of things? I get timeouts 100% of the time.

@megan07 how do you increase the timeout? I see no config for that in the provider: https://registry.terraform.io/providers/hashicorp/googleworkspace/latest/docs