Support Block Kit with Slack notifications

[mambax]

Hi Prometheus

Currently <slack-config> is using Simple formatting.

This is marked by Slack themselves as deprecated (or at least most hints dropped say: Format using Block Kit).

The new way to go seems to be Block-Kit.

I don't want to ask "Is there a plan?" as this would be a question. I'd like to pose as feature request to add something like:

<slack_config_blocks> which would then accept stuff like:

And configuration like so (I am new to Yaml and Go Tmpl):

blocks:
- type: section
  text:
    type: mrkdwn
    text: ":fire: `Critical` alert in *production*: `ELK running out of shards (2
      left)`"
- type: section
  fields:
  - type: mrkdwn
    text: |-
      *Value before:*
      22
  - type: mrkdwn
    text: |-
      *Value now:*
      2
  - type: mrkdwn
    text: |-
      *Action:*
      Contact operations.
  - type: mrkdwn
    text: |-
      *Outcome:*
      X will happen.
- type: actions
  elements:
  - type: button
    text:
      type: plain_text
      emoji: true
      text: Victorops
    style: primary
    value: click_me_123
  - type: button
    text:
      type: plain_text
      emoji: true
      text: Silence
    style: danger
    value: click_me_123

Obvisouly my hardcoded strings would be replaced by {{ .alertProperties }}

Adding this I believe would make our DevOps plans (and hopefully others) dreams come true ❤️

[simonpasquier]

I'm worried that the configuration schema might quickly get out of control. But there's nothing preventing someone to try to come up with something reasonable.

[vanugrah]

Block kit makes formatting of messages SO much cleaner and introduces several new types to play around with. If I could +1000 I would :)

[KarthikRangaraju]

+1 The possibilities are endless. These are all the things we can do: https://app.slack.com/block-kit-builder/T091CRSGH#%7B%22blocks%22:%5B%5D%7D

[niko2]

Simple Formatting seems to be broken on Slack Desktop (at least for buttons). Since there is no more Slack support, this formatting could be completely broken soon.

[tom0010]

Also looking forward to this for better slack interactivity with alertmanager. Are there any updates maybe?

[johanfleury]

Hey, I’m willing to make a PR for that.

I see two ways of doing that:

• blocks field in slack_config is a tmpl_string that must render the JSON object sent to slack:

blocks: >
  {
    "type": "header",
    "text": {
      "type": "plain_text",
      "text": "{{ template "slack.default.title" . }}",
      "emoji": true
    }
  },
  {{ range .Alerts.Firing }}
  {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Alert:* '{{ .Annotations.summary }} - `{{ .Labels.severity }}`'\n*Description:* '{{ .Annotations.description }}'"
    }
  }, # Find a way to deal with comma after last block
  {{ end }}

• blocks field in slack_config is a complex object that replicate Slack API with added options like repeat_for:

blocks:
  - type: header
    text:
      type: plain_text
      text: '{{ template "slack.default.title" . }}'
  - type: section
    text:
      type: mrkdwn
      text: >-
        *Alert:* '{{ .Annotations.summary }} - `{{ .Labels.severity }}`
        *Description:* '{{ .Annotations.description }}'
      repeat_for: firing # Could be firing, resolved or all

As an administrator I prefer option 2, I find it clearer, but it’s way more work to implement it and it’s less flexible than option 1.

What do you think?

[tom0010]

I think it would make sense to go with the yaml formatting, just like @mambax said. With @johanfleury's option 2, it's like yaml, but doesn't seem to be valid as such. The comma's are not needed, hence why I said about @mambax's comment.

[johanfleury]

It was just bad copy/paste, I fixed the snippets.

[IppX]

@johanfleury I like your first approach for 3 reasons:

• as a user of this feature, I'll need to get familiar with the slack API and block features anyway. The best way to do so is to use the build kit viewer and get my formatting straight first and then I can just copy paste it here and add the necessary golang templating.
• it's more flexible regarding slack API changes.
• since it's easier to implement it would enable us to have the feature earlier.

[tom0010]

@IppX you are right about the JSON formatting being similar to the block kit builder, however there would be loops and variables in there payload anyway so you would still have to edit the payload, copy and paste would be invalid. blocks is also a list, whereas the JSON version does not really account and inform the user about it being a list, the YAML version does. Option 2 also keeps the formatting the same, the Prometheus config is actually in YAML, so it would mean we would have a JSON string inside of a YAML file. IMO it makes more sense to keep the whole YAML file, in YAML. There are also various tools that do JSON to YAML conversion online, so that isn't really a problem regarding the block kit builder.

[johanfleury]

I started working on option 2 and I’m facing a bunch of issues (explaining below) and I think option 1 is a better option if we want to keep code clean.

Option 2 also keeps the formatting the same, the Prometheus config is actually in YAML, so it would mean we would have a JSON string inside of a YAML file.

I definitely agree with that, I’d also rather have everything in YAML, but as I suspected when writing my first comment, this is a lot of work.

---

Now, regarding the issues I’m facing with option 2.

I don’t want to have to rewrite every block types, so I started creating a “generic” block type:

type SlackConfig struct {
	// ...
	Blocks []*SlackBlock `yaml:"blocks,omitempty" json:"blocks,omitempty"`
	//...
}

type SlackBlock struct {
	RepeatFor string                 `yaml:"repeat_for,omitempty" json:"repeat_for,omitempty"`
	Config    map[string]interface{} `yaml:"config,omitempty" json:"config,omitempty"`
}

This would translate to a config that looks like that:

slack_configs:
  - ...
    blocks:
      - config:
          type: header
          text:
            type: plain_text
            text: '{{ template "slack.default.title" . }}'
      - repeat_for: firing # Could be firing, resolved or all
        config:
          type: section
          text:
            type: mrkdwn
            text: >-
              *Alert:* '{{ .Annotations.summary }} - `{{ .Labels.severity }}`
              *Description:* '{{ .Annotations.description }}'

Problem is, when go-yaml is parsing that config it unmarshals any maps (e.g. config.text) as a map[interface{}]interface{} which Go’s encoding/json cannot marshal back to json (see https://go.dev/play/p/486JtJW9Sjw).

This also brings issue with templating as we now need to recurse over the whole map to find any string value and apply the templating function.

Also, template’s context inside blocks with repeat_for will be different to any other templates (you would only have access to Alert’s fields and not the whole data).

To be honest, I’m really not sure if there’s a way to keep both the code and the config clean at the same time.

I’ll create a draft PR for this option if someone wants to look at it, but I feel like it’s too messy and a bit too much work.

[lololozhkin]

Yes, my team is waiting for this too. Simple formatting is deprecated, and not all messages are rendered correctly.

For example: we cannot use emojis in the title of the message, but that will be very useful, to show severity of alert. Interesting thing, that this functionality is working on PC and on Android, but not on iphones.

That is not a single issue, but just for example, to show, that simple formatting is really deprecated.

My team contacted Slack help, and they said, that we need to use blocks for sending alerts. We're waiting for news from this issue!

[anthonator]

Anybody know if it’s possible to use Block Kit using Alertmanager http_ config and Slack incoming webhooks? This might be a better approach than a direct Slack integration.

[marcusramberg]

Anybody know if it’s possible to use Block Kit using Alertmanager http_ config and Slack incoming webhooks? This might be a better approach than a direct Slack integration.

Did you ever try this? Seems like it would be a good alternative for rich alerts.

[grobinson-grafana]

I prefer Option 2 over Option 1 as I have some experience using Jinja templates to output Yaml, and found writing templates that produced valid Yaml quite difficult when using if statements and for loops. For example, indentation. I am sure we can solve the issue with de-serialization, and having a Go struct for each kind of block would allow us to validate the configuration at start up time.

[grobinson-grafana]

Here is an example of how I think it could look. However, for this to work, we also need to change UnmarshalStrict to Unmarshal in Load:

// config/config.go
// Load parses the YAML input s into a Config.
func Load(s string) (*Config, error) {
	cfg := &Config{}
	err := yaml.Unmarshal([]byte(s), cfg)
        ...
}

// config/notifiers.go
type SlackBlock struct {
	Type string `yaml:"type,omitempty" json:"type,omitempty"`
	// Config contains the configuration for the block. For example, if this
	// is a header block then Config is a *SlackBlockHeader.
	Config interface{} `yaml:"-" json:"-"`
}

type SlackBlockHeader struct {
	Type string               `yaml:"type,omitempty" json:"type,omitempty"`
	Text SlackBlockHeaderText `yaml:"text,omitempty" json:"text,omitempty"`
}

type SlackBlockHeaderText struct {
	Type string `yaml:"type,omitempty" json:"type,omitempty"`
	Text string `yaml:"text,omitempty" json:"text,omitempty"`
}

func (c *SlackBlock) UnmarshalYAML(unmarshal func(interface{}) error) error {
	type plain struct {
		Type   string      `yaml:"type,omitempty" json:"type,omitempty"`
		Config interface{} `yaml:"-" json:"-"`
	}
	if err := unmarshal((*plain)(c)); err != nil {
		return err
	}
	switch ty := c.Type; ty {
	case "header":
		var header SlackBlockHeader
		if err := unmarshal(&header); err != nil {
			return err
		}
		c.Config = header
	default:
		return fmt.Errorf("unknown block: %s", ty)
	}

	return nil
}

[alkinks]

Yes, my team is waiting for this too. Simple formatting is deprecated, and not all messages are rendered correctly.

For example: we cannot use emojis in the title of the message, but that will be very useful, to show severity of alert. Interesting thing, that this functionality is working on PC and on Android, but not on iphones.

That is not a single issue, but just for example, to show, that simple formatting is really deprecated.

My team contacted Slack help, and they said, that we need to use blocks for sending alerts. We're waiting for news from this issue!

+1 for the issue. I'm facing a very similar issue to this - I have a nicely templated slack message with 4 buttons, which all show up in the phone app, but only 3 show up in the desktop app, or web. I had a similar response from Slack support.

[blacksails]

There is also a third option - Represent the slack API in yaml and make it a template. Like option 1, let blocks be a template but in yaml. Encode to json before sending it to slack. Downside: you cant copy/paste json from the slack block builder tool.

I dislike having to template JSON, which would be required in option 1. It is easier to template yaml.

Option 2, will probably have annoying edge cases, eg. what would a nested loop look like. Almost feels like you either would need to design a templating DSL in yaml or keep it simple but inflexible. Option 1 or the above proposed solution is better IMO.