terraform-provider-grafana

terraform-provider-grafana copied to clipboard

Terraform Version

• Terraform: 1.1
• Terraform Grafana Provider: 1.23.0
• Grafana: Cloud 8.5.2

Affected Resource(s)

• grafana_synthetic_monitoring_check

Expected Behavior

Multiple synthetic monitoring checks are created.

Actual Behavior

Errors during the creation of the checks indicating deadlocks due to the speed that the monitoring checks are created.

│ Error: check add request: status="409 Conflict", msg="Failed to add check to database", err="inserting probes for check 254454: inserting probe 19 for check 254454: Error 1213: Deadlock found when trying to get lock; try restarting transaction"
│ 
│   with grafana_synthetic_monitoring_check.api_http["<snip>"],
│   on monitoring.tf line 184, in resource "grafana_synthetic_monitoring_check" "api_http":
│  184: resource "grafana_synthetic_monitoring_check" "api_http" {

Steps to Reproduce

1. Add ~30 grafana_synthetic_monitoring_check resources to Terraform.
2. terraform apply against a Grafana Cloud instance.

Important Factoids

This is more an issue with Grafana Cloud handling the creation so I'm not sure if this is the most appropriate place to report this issue, but figure rate-limiting could be added to the provider or retry logic implemented.

Annoyingly the 409 Conflict returned by the API makes it slightly harder as it's the same error code if a duplicate check is added.

any update on this?

Workaround: Limit the number of concurrent operation to 1.

when applying from terraform CLI, setting TF_CLI_ARGS_apply="-parallelism=1" envirement variable will limit parallelism to 1 and terraform will create the checks one by one.

for more details, see docs

The fact that the error message out of the API talks about a deadlock is unfortunate (we are bubbling the error from the database up the stack). You are right, this is something we need to fix in the API.

@indygriffiths, we deployed a change in the API, this is not longer happening. Apologies for the trouble.