Support for env vars when using GrafanaNotificationChannel

[eduardobaitello]

I'm creating a GrafanaNotificationChannel with slack type, which has a sensitive Webook URL configuration.

I'm trying to figure out how to use a secret + envVar for the Webhook URL, similar to supported for datasources.

The grafana container has $SLACK_WEBHOOOK_URL exported as an environment variable, but the Webhook URL ends as a literal string (I think because it's inside spec.json).

Is there a feasible way to do this right now?

apiVersion: integreatly.org/v1alpha1
kind: GrafanaNotificationChannel
metadata:
  name: slack-notification-channel
spec:
  name: slack-notification-channel
  json: >
    {
      "uid": "slack-channel-notification",
      "name": "Slack Channel Notificaton",
      "type": "slack",
      "isDefault": false,
      "sendReminder": false,
      "disableResolveMessage": false,
      "frequency": "5m",
      "settings": {
        "addresses": "",
        "autoResolve": true,
        "httpMethod": "POST",
        "icon_emoji": ":grafana:",
        "icon_url": "",
        "mentionGroups": "",
        "mentionUsers": "",
        "recipient": "#slack-notification-channel",
        "severity": "critical",
        "token": "",
        "uploadImage": true,
        "url": "$SLACK_WEBHOOOK_URL",
        "username": "Grafana Notification"
      },
      "secureFields": {
        "url": true
      }
    }

[eduardobaitello]

Furthermore, I just realized that when declaring the URL explicitly,

json: >
 {
 [...]
      "url": "https://hooks.slack.com/xxxxx/xxxxx/xxxx",
       "username": "Grafana Notification"
     },
     "secureFields": {
       "url": true
     }
 }

The GrafanaNotificationChannel is created without encryption, and the Webhook value becomes exposed on the UI. Once opening the UI and clicking on the Save button, the value becomes encrypted.

I think that this CRD will need support for something like secureJsonData to overcome this (maybe this secure_settings config).

[hubeadmin]

Hi @eduardobaitello Thanks for raising this, currently the notificationchannel feature only supports raw json as of now (for simplicity), your suggestion definitely makes sense, This would require adding some new structs to the notificationchannel struct from under https://github.com/grafana-operator/grafana-operator/blob/master/api/integreatly/v1alpha1/grafanadatasource_types.go#L66

[weisdd]

@eduardobaitello

Env expansion

Datasources are currently supplied to Grafana via a configuration file, that's why environment expansion works there: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#env-provider

Notification channels are configured directly via API. Grafana doesn't seem to support expansion there. In fact, if you go to Grafana UI, pick webhook notifications and try to set a password there to something like $SLACK_WEBHOOOK_URL or ${SLACK_WEBHOOOK_URL} (assuming this env is available within the grafana container), the password will be sent to a webhook by grafana as is. You can test that yourself with a simple Go code:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func authorizationHandler(w http.ResponseWriter, r *http.Request) {
	header := r.Header.Get("Authorization")
	fmt.Printf("Authorization: %v\n", header)
}

func main() {
	http.HandleFunc("/", authorizationHandler)

	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(err)
	}
}

Safe Slack webhook configuration

In Chrome-based browsers, you can open developer tools (F12). There, you could explore Grafana's requests in the Network section. It's always helpful to go there in case you're not really sure about the correct json structure. In the end, both Grafana UI and grafana-operator rely on the same set of APIs.

As you can see on the screenshot, url and token should be placed within secureSettings, not within settings. And secureFields is empty.

Here's the CR spec that you were looking for:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaNotificationChannel
metadata:
  name: pager-duty-channel
  labels:
    app.kubernetes.io/instance: grafana-operator
spec:
  name: pager-duty-channel.json
  json: >
    {
      "uid": "slack-channel-notification",
      "name": "Slack Channel Notificaton",
      "type": "slack",
      "isDefault": false,
      "sendReminder": false,
      "disableResolveMessage": false,
      "frequency": "5m",
      "settings": {
        "addresses": "",
        "autoResolve": true,
        "httpMethod": "POST",
        "icon_emoji": ":grafana:",
        "icon_url": "",
        "mentionGroups": "",
        "mentionUsers": "",
        "recipient": "#slack-notification-channel",
        "severity": "critical",
        "uploadImage": true,
        "username": "Grafana Notification"
      },
      "secureSettings": {
        "token": "",
        "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }

[weisdd]

@HubertStefanski I think there's no need for any enhancements here as the operator already supports the desired configuration.

[eduardobaitello]

I could be wrong, but I had already tested this. I'll try again and let you know here, thanks!

But using your example, anyway I wouldn't be able to set the webhook as a secret since I can't use it as a variable, leading the webhook to be exposed on the GrafanaNotificationChannel object 😔

[weisdd]

@eduardobaitello But what makes you concerned about the webhook url in the CR? In the end, normally, it's only platform team who has enough rights to query those CRs from k8s API-server. The same goes for the Grafana CR - if someone has access to it, then that person can potentially see admin password or OIDC client secret, depending on your setup.

• Grafana itself does not support the expansion.
• Forcing operator to go through secrets referenced in grafana deployment and expand those when making requests to certain APIs is possible, but would overcomplicate the code (operator would have to watch for changes in the secrets + logic around that). Plus, if end users are allowed to freely create GrafanaNotificationChannel, they can force grafana to send the webhook url to a malicious webhook.
• There's a fork that does expansion through envs available to operator itself: https://github.com/infobloxopen/grafana-operator/pull/67, though, again, with unrestricted access to GrafanaNotificationChannel CRs, it means that someone can potentially force grafana to send credentials to a malicious webhook.

So, I'd still think that Kubernetes RBAC is the answer.

[eduardobaitello]

@weisdd thanks for the clarification.

I got your point, and it really makes sense.

I just tested as you suggested, and the Grafana Notification was created with the url encrypted in the Grafana database.

This is enough for what I need, so I am closing this issue. Thanks again! ❤️