Add retry util func with polling interval

[kevinsi4508]

We currently have an util to retry something with a timeout: retryTimeDuration().

This PR adds retryWithPolling() to allow retrying with a timeout and a polling interval.

If this PR is for Terraform, I acknowledge that I have:

• [x] Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• [x] Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• [x] Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• [x] Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• [x] Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @melinath, a repository maintainer, has been assigned to assist you and help review your changes.

:question: First time contributing? Click here for more details

---

Your assigned reviewer will help review your code by:

• Ensuring it's backwards compatible, covers common error cases, etc.
• Summarizing the change into a user-facing changelog note.
• Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

---

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 125 insertions(+)) Terraform Beta: Diff ( 2 files changed, 125 insertions(+)) TF Validator: Diff ( 3 files changed, 33 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2176 Passed tests 1936 Skipped tests: 238 Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccComputeInstance_soleTenantNodeAffinities

[modular-magician]

Tests passed during RECORDING mode: TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode: TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log]

Please fix these to complete your PR View the build log or the debug log for each test

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 126 insertions(+)) Terraform Beta: Diff ( 2 files changed, 126 insertions(+)) TF Validator: Diff ( 3 files changed, 34 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2176 Passed tests 1935 Skipped tests: 238 Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccComputeInstance_soleTenantNodeAffinities|TestAccCGCSnippet_eventarcWorkflowsExample

[modular-magician]

Tests passed during RECORDING mode: TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode: TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log] TestAccCGCSnippet_eventarcWorkflowsExample[Error message] [Debug log]

Please fix these to complete your PR View the build log or the debug log for each test

[ScottSuarez]

Do you have context here? Seems there is no consumer. Why is this necessary? I worry will hit api's get quotas quicker, especially if a lot of resources of a certain type are created/called.

[kevinsi4508]

Good point!

I talked to Riley a bit about this PR. The built-in retry with backoff logic doesn't backoff long enough because there is a max of 10s. This utility function provides a way to backoff with an a poll interval greater than 10s. I am planning to set the poll interval to 30s for Cloud Bigtable GC policy (something the server suggests in the response). Without this long poll interval users can run out of per minute quota quickly when there are multiple GC policies need to be added at the same time.

See this internal bug: b/247584824 for more context.

[kevinsi4508]

PTAL