401 body error on next Terraform plan (after successfully creating dashboard)

[slime-uk]

Terraform Version

• Terraform: 1.2.0
• Terraform AzureRM Provider: 3.4.0
• Terraform AzAPI Provider: 0.4.0
• Terraform Grafana Provider: 1.25.0
• Grafana: 9.0.1 (Azure managed grafana preview feature)

Affected Resource(s)

• grafana_dashboard

Terraform Configuration Files

resource "azurerm_resource_group" "grafana_rg" {
    name     = var.grafana_rg_name
    location = var.location
}

# And next a storage account
resource "azurerm_storage_account" "grafana_sa" {
    name                     = lower("${var.prefix}${var.grafana_sa_name_suffix}")
    resource_group_name      = azurerm_resource_group.grafana_rg.name
    location                 = azurerm_resource_group.grafana_rg.location
    account_tier             = "Standard"
    account_replication_type = "LRS"
}

# Now the actual grafana managed instance
# TODO: This will be replaced with AzureRM provider once it is available
resource "azapi_resource" "grafana" {
    type           = "Microsoft.Dashboard/grafana@2021-09-01-preview"
    name            = "${var.prefix}${var.grafana_name_suffix}"
    parent_id       = azurerm_resource_group.grafana_rg.id

    body = jsonencode({
        location   = azurerm_resource_group.grafana_rg.location
        properties = {
            zoneRedundancy                      = "Disabled"
            autoGeneratedDomainNameLabelScope   = "TenantReuse"
        }
        sku = {
            name                                = "Standard"
        }
        identity = {
            type                                = "SystemAssigned"
        }
    })
  
    response_export_values = ["properties.endpoint", "identity.principalId"]
}

###################################################################################################
# GRAFANI MI/SP ACCESS:
# The document says "By default, when a Grafana workspace is created, Azure Managed Grafana grants it the Monitoring Reader role for all Azure Monitor data and Log Analytics resources within a subscription."
# but it seems not set so far, so it'll assign it.
# https://docs.microsoft.com/en-us/azure/managed-grafana/how-to-permissions
resource "azurerm_role_assignment" "grafana_ra" {
    scope                   = "/subscriptions/${data.azurerm_client_config.current.subscription_id}"
    role_definition_name    = "Monitoring Reader"
    principal_id            = jsondecode(azapi_resource.grafana.output).identity.principalId
}
# We also want the Grafana MI/SP to be a grafana admin
resource "azurerm_role_assignment" "grafana_ra_owner_sami" {
    scope                   = azapi_resource.grafana.id
    role_definition_name    = "Grafana Admin"
    principal_id            = jsondecode(azapi_resource.grafana.output).identity.principalId
}
# And access to the storage account?
resource "azurerm_role_assignment" "grafana_ra_owner_sami_storageblobdc" {
    scope                   = azurerm_storage_account.grafana_sa.id
    role_definition_name    = "Storage Blob Data Contributor"
    principal_id            = jsondecode(azapi_resource.grafana.output).identity.principalId
}
resource "azurerm_role_assignment" "grafana_ra_owner_sami_storagefilec" {
    scope                   = azurerm_storage_account.grafana_sa.id
    role_definition_name    = "Storage File Data SMB Share Contributor"
    principal_id            = jsondecode(azapi_resource.grafana.output).identity.principalId
}

# Assign the TF SP the admin rights to grafana - as we need permissions to administer it via API calls
resource "azurerm_role_assignment" "grafana_ra_owner_tf_sp" {
    scope                   = azapi_resource.grafana.id
    role_definition_name    = "Grafana Admin"
    principal_id            = var.grafana_token_object_id
}
# And access to the storage account?
resource "azurerm_role_assignment" "grafana_ra_owner_tf_sp_storageblobdc" {
    scope                   = azurerm_storage_account.grafana_sa.id
    role_definition_name    = "Storage Blob Data Contributor"
    principal_id            = var.grafana_token_object_id
}
resource "azurerm_role_assignment" "grafana_ra_owner_tf_sp_storagefilec" {
    scope                   = azurerm_storage_account.grafana_sa.id
    role_definition_name    = "Storage File Data SMB Share Contributor"
    principal_id            = var.grafana_token_object_id
}

###################################################################################################
# Get API token
# We want this to run on every TFC plan/apply to ensure agent /scripts/get_grafana_token.sh script is chmod +x
resource "null_resource" "set_permissions_to_script" {
    triggers = {
        time_last_run = timestamp()
    }

    # Set the required script to be executable and readable
    provisioner "local-exec" {
        interpreter = ["/bin/bash", "-c"]
        command = <<EOH
            chmod 755 ${path.module}/scripts/get_grafana_token.sh
            ls -al ${path.module}/scripts    
        EOH
    }
}

data "external" "grafanatoken" {
    program = ["bash", "-c", "${path.module}/scripts/get_grafana_token.sh"]

    query = {
        # arbitrary map from strings to strings, passed to the external program as the data query.  jq reads these as .name and case is sensitive!
        clientid = var.grafana_token_client_id
        clientsecret = var.grafana_token_client_secret
        tenantid = data.azurerm_client_config.current.tenant_id
    }

    depends_on  = [
        azurerm_role_assignment.grafana_ra,
        azurerm_role_assignment.grafana_ra_owner_sami, 
        azurerm_role_assignment.grafana_ra_owner_tf_sp, 
        null_resource.set_permissions_to_script,
    ]
}

###################################################################################################
# Now do something with the grafana instance using the grafana provider and API (we need the access token from earlier)
# Configure provider to grafana API with aquired access token (which expires)
provider "grafana" {
    alias                   = "base"
    url                     = jsondecode(azapi_resource.grafana.output).properties.endpoint
    auth                    = data.external.grafanatoken.result["token"]
    org_id                  = 1

    # Set to true if you want to save only the sha256sum instead of complete dashboard JSON in TF state
    store_dashboard_sha256  = true
}

# Pull a dashboard from grafana
data "http" "dashboard_azure_monitor_storage_insights" {
    url    = "https://grafana.com/api/dashboards/14469/revisions/1/download"
    request_headers = {
        Accept = "application/json"
    }
}
# Use the body to create a template for our new dashboard
data "template_file" "dashboard_azure_monitor_storage_insights" {
    template = data.http.dashboard_azure_monitor_storage_insights.body
    vars = {
        DS_AZURE_MONITOR = "azure-monitor-oob"
        VAR_NS           = "Microsoft.Storage/storageAccounts"
    }
}
# Then add the dashboard using the API
resource "grafana_dashboard" "azure_monitor_storage_insights" {
    provider    = grafana.base
    config_json = data.template_file.dashboard_azure_monitor_storage_insights.rendered
    overwrite   = true
}

Debug Output

https://gist.githubusercontent.com/slime-uk/8fa1e07038f9e0a9f2a17f1459ed1b5d/raw/ea7a3832cb1c3e8f0d778d61fc9fe5914266e597/gistfile1.txt

Expected Behavior

A managed dashboard should not cause the next TF plan to fail - what am I doing wrong?

Actual Behavior

The dashboard creation works fine but I can never TF plan again as I simply receive 401 body error:

Error: status: 401, body:
with grafana_dashboard.azure_monitor_storage_insights
on main.tf line 334, in resource "grafana_dashboard" "azure_monitor_storage_insights":
resource "grafana_dashboard" "azure_monitor_storage_insights" {

This continues until I remove the managed dashboard from TF state using terraform state rm command

Steps to Reproduce

1. Create an Azure RG, Azure storage account, Assign role permissions, then create an Azure managed grafana (uisng AzAPI provider) and then after obtaining API token, create a dashboard using JSON from here: https://grafana.com/api/dashboards/14469/revisions/1/download (also tried with later revisions and raw json, and a different dashboard - all with same 401 body result on 2nd plan). Everything works - then re-plan with no changes - error 401 body results.

Important Factoids

We use Terraform Cloud and note that we are using a preview feature of Azure Managed Grafana and so have to use AzAPI terraform provider to do the actual creation of the managed grafana (Terraform AzureRM provider is yet to support this resource). I am then getting an API token and using that as the auth field for the Terraform grafana provider - similar to this example: https://gist.github.com/ToruMakabe/64cd2db6e7a886047d08b6d45a2050ec. On each run, I obtain a new token - is this the issue?

Thank you.