hashicorp
Should back off and retry on 429
We're starting to see HTTP 429 Too many requests on TFE intermittently, which then breaks the run. The provider should back off gracefully and retry when this happens.
Running from within Terraform Cloud, if that is relevant.
Hi @carlpett,
The library used by the provider to talk to TFE already has a back off mechanism that also accounts for throttling.
Can you please run terraform again with the environment variable TF_LOG set to DEBUG and share the logs with us?
Hi @koikonom, I haven't seen this for a while (but haven't been pushing that workspace very hard on the other hand). There's an internal ticket (#25285) with quite a bit of info and some links to failed runs, if that helps?
Unfortunately I am not part of the TFC team so I cannot access any of these links.
If I had to guess I'd say that probably this is a combination of managing multiple workspaces (resources) and terraform's default parallelism setting (10).
The only suggestion I can offer from a provider perspective would be to try setting the parallelism parameter for terraform applies to something smaller than the default.
See here how you can do that.
Closing this issue because the problem of hitting throttle limits is fairly well known in terraform and workarounds have been discussed elsewhere, such as using smaller state files and less parallelism. I don't think this is a specific limitation of this provider, and the underlying http client is already making retry/fallback efforts.
