Re-applying a dashboard resource quickly after destroying it can result in `403 Access denied to this dashboard`

[eamonryan]

Terraform Version

• Terraform: v1.3.1
• Terraform Grafana Provider: v1.29.0
• Grafana: 9.1.6

Affected Resource(s)

• grafana_dashboard

Terraform Configuration Files

Any TF that applies dashboards should work e.g. the example at https://grafana.com/docs/grafana-cloud/infrastructure-as-code/terraform/dashboards-github-action/

Expected Behavior

I performed a terraform destroy to delete several dashboards, then a terraform apply to re-create them right afterward. I expected this to succeed.

Actual Behavior

I received 403 Forbidden messages for the dashboards. I re-tried 30s or so later and it worked fine.

Destroy:

grafana_dashboard.dashboards["Folder3/heynow3.json"]: Destroying... [id=KJkEAHV43]
grafana_dashboard.dashboards["Folder2/heynow2.json"]: Destroying... [id=KJkEAHV42]
grafana_dashboard.dashboards["Folder3/heynow3b.json"]: Destroying... [id=KJkEAHV3b]
grafana_dashboard.dashboards["Folder2/heynow2b.json"]: Destroying... [id=KJkEAHV2b]
grafana_dashboard.dashboards["Folder1/heynow1b.json"]: Destroying... [id=KJkEAHV1b]
grafana_dashboard.dashboards["Folder1/heynow.json"]: Destroying... [id=KJkEAHV4k]
grafana_dashboard.dashboards["Folder2/heynow2b.json"]: Destruction complete after 1s
grafana_dashboard.dashboards["Folder1/heynow.json"]: Destruction complete after 1s
grafana_dashboard.dashboards["Folder1/heynow1b.json"]: Destruction complete after 1s
grafana_dashboard.dashboards["Folder3/heynow3.json"]: Destruction complete after 1s
grafana_dashboard.dashboards["Folder3/heynow3b.json"]: Destruction complete after 1s
grafana_dashboard.dashboards["Folder2/heynow2.json"]: Destruction complete after 1s
grafana_folder.folders["Folder2"]: Destroying... [id=28]
grafana_folder.folders["Folder3"]: Destroying... [id=27]
grafana_folder.folders["Folder1"]: Destroying... [id=29]
grafana_folder.folders["Folder2"]: Destruction complete after 0s
grafana_folder.folders["Folder1"]: Destruction complete after 0s
grafana_folder.folders["Folder3"]: Destruction complete after 0s

Apply:

grafana_folder.folders["Folder2"]: Creating...
grafana_folder.folders["Folder1"]: Creating...
grafana_folder.folders["Folder3"]: Creating...
grafana_folder.folders["Folder2"]: Creation complete after 2s [id=38]
grafana_folder.folders["Folder3"]: Creation complete after 2s [id=37]
grafana_folder.folders["Folder1"]: Creation complete after 2s [id=36]
grafana_dashboard.dashboards["Folder2/heynow2.json"]: Creating...
grafana_dashboard.dashboards["Folder1/heynow1b.json"]: Creating...
grafana_dashboard.dashboards["Folder1/heynow.json"]: Creating...
grafana_dashboard.dashboards["Folder3/heynow3.json"]: Creating...
grafana_dashboard.dashboards["Folder3/heynow3b.json"]: Creating...
grafana_dashboard.dashboards["Folder2/heynow2b.json"]: Creating...
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder1/heynow.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {
│ 
╵
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder2/heynow2b.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {
│ 
╵
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder2/heynow2.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {
│ 
╵
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder3/heynow3.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {
│ 
╵
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder3/heynow3b.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {
│ 
╵
╷
│ Error: status: 403, body: {"message":"Access denied to this dashboard"}
│ 
│   with grafana_dashboard.dashboards["Folder1/heynow1b.json"],
│   on dashboards.tf line 59, in resource "grafana_dashboard" "dashboards":
│   59: resource "grafana_dashboard" "dashboards" {

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Apply one or more dashboards via terraform
2. Run terraform destroy to delete them
3. Re-run terraform apply within a short time period (5-10s) to re-apply them
4. See if 403 occurs, I'm unsure if it will happen every time, but I saw it happen at least twice in the past couple days.

Important Factoids

Are there anything atypical about your accounts that we should know? Running terraform directly from a laptop against a Grafana deployment in a GKE cluster. I was port-forwarding the Grafana pod.

[julienduchesne]

Just a note: I tried really hard to reproduce this one locally, without success. It may be a port-forwarding issue which means it will be hard to add an automated test for it, if we find the cause

[julienduchesne]

Will close this because the provider has changed an awful lot since then and I'm still unable to reproduce. Feel free to re-open if this is still an issue, I will look into it more!