[grafana] Alert Provisioning - failure to parse file contact_points.yaml: yaml: unmarshal errors:

[Matt-Holeman]

Contact Points File

apiVersion: 1
contactPoints:
  - name: cp_name
    receivers:
      - uid: cp_name_uid
        type: webhook
        settings:
          httpMethod: POST
          url: ${grafana_alert_endpoint}
          username: admin
          password: ${grafana_alert_endpoint_key}

Terraform Helm Release Snippet

This is how we adding any alerting related files. We currently have these split in contact_points, notification_policies, and alert_rules but intend to experiment with joining in the same config file. And then loading via configmap, similar to dashboards/datasources, once that becomes available. I understand that this can be refined, we're just making sure 9.1 is working for our environment at the moment, migrating from 8.x with legacy alerting.

# Helm chart doesn't currently support (As of Sept 2022) auto-detection of an alerting config map so need to pass in through here.
    alerting = merge(
      {
        for filename in fileset("${path.module}/grafana/alerting", "{*.yaml}") : "${filename}" => file("${path.module}/grafana/alerting/${filename}")
      },
      {
        for filename in fileset("${path.module}/grafana/alerting", "{*.json}") : "${filename}" => file("${path.module}/grafana/alerting/${filename}")
      },
      {
        for filename in fileset("${path.module}/grafana/alerting", "{*.tftpl}") : "${replace(filename, ".tftpl", "")}" => templatefile("${path.module}/grafana/alerting/${filename}",
          {
            grafana_alert_endpoint     = var.grafana_alert_endpoint
            grafana_alert_endpoint_key = var.grafana_alert_endpoint_key
            environment                       = var.environment_name
          }
        )
      }
    )

When attempting to deploy version 6.38.6 of the grafana helm chart using the above contact points file, I get the below error in the grafana container.

Failed to start grafana. error: failure to parse file contact_points.yaml: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `apiVers...` into alerting.AlertingFileV1
failure to parse file contact_points.yaml: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `apiVers...` into alerting.AlertingFileV1

This was not present in versions of the helm chart prior to this specific version (which is why I'm raising here because 6.38.5 and 6.38.6 are both the same app version).

We're attempting to make use of Grafana 9.1 unified alerting and the recent provisioning functionality, we need to use a later version than something like 6.33.1 in the hope they fix issues with alert math expressions making a value of $A > 75 not functional (see: https://github.com/grafana/grafana/issues/54532). Which in turn we apparently need to use instead of classic expressions to be able to make use of multi-dimensional alerts.

Has the expected syntax of the file changed or requirements on deployment/formatting changed as part of this version?

[zanhsieh]

@Matt-Dev87 Is it because the alert now support helm function? https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function

[Matt-Holeman]

@Matt-Dev87 Is it because the alert now support helm function? https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function

That could be something we look to do if the terraform templating (how we're passing in the values at the moment) becomes more awkward with values/escaping - It's my understand that using it in the way we have should result in helm only receiving a full yaml file with the endpoint values populated.

That being said, I have tried this with removing any templating from the contact_points.yml file as well as not passing it through the terraform templatefile function.

[gittyabhay]

Hi is there any solution to this, I am trying to add alert rules using file provisioning, Error: error="failure to parse file Export_alert_rule.yaml: invalid duration 7.776e+06"