'mimirtool alertmanager get' output is invalid and messy

[adeverteuil]

Describe the bug

When I run mimirtool alertmanager get, the output is not very clean.

Problems:

• The Alertmanager config is at the root of the printed YAML output, along with a Templates key.
• The Templates: key is capitalized, it should be lowercase, for uniformity.
• The templates data is not indented, this YAML is invalid.
• The Alertmanager config should be under its own key as well.

Requested fixes:

• There should be two root dictionary keys alertmanager_config and template_files, to be consistent with the POST body requirements.
  https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/#set-alertmanager-configuration
• The output of mimirtool alertmanager get should be usable as an input for mimirtool alertmanager load.
• Returning the response of the GET /api/v1/alerts API call without any processing would be better.

Here is an example of what I get:

alex@lgtm-lab-01:~$ mimirtool alertmanager get
global:
    http_config:
        follow_redirects: true
        proxy_url: null
        tls_config:
            insecure_skip_verify: false
    opsgenie_api_url: https://api.opsgenie.com/
    pagerduty_url: https://events.pagerduty.com/v2/enqueue
    resolve_timeout: 5m
    smtp_hello: localhost
    smtp_require_tls: true
    smtp_smarthost: ""
    victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
    wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
receivers:
    - name: no-op
    - name: logger
      webhook_configs:
        - http_config:
            follow_redirects: true
            proxy_url: null
            tls_config:
                insecure_skip_verify: false
          max_alerts: 0
          send_resolved: true
          url: http://localhost:8888/gem-alertmanager
route:
    receiver: logger
    repeat_interval: 5m
templates:
    - test

Templates:
test:
{{ define "test" }}
  asdf
{{ end }}

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir v2.2.0 (tested on GEM v2.2.0).
2. Create a tenant with an Alertmanager config.
3. Use mimirtool alertmanager get to read the Alertmanager config.

Expected behavior

Here is the fixed output I would prefer to see:

alertmanager_config: |
    global:
        http_config:
            follow_redirects: true
            proxy_url: null
            tls_config:
                insecure_skip_verify: false
        opsgenie_api_url: https://api.opsgenie.com/
        pagerduty_url: https://events.pagerduty.com/v2/enqueue
        resolve_timeout: 5m
        smtp_hello: localhost
        smtp_require_tls: true
        smtp_smarthost: ""
        victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
        wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
    receivers:
        - name: no-op
        - name: logger
          webhook_configs:
            - http_config:
                follow_redirects: true
                proxy_url: null
                tls_config:
                    insecure_skip_verify: false
              max_alerts: 0
              send_resolved: true
              url: http://localhost:8888/gem-alertmanager
    route:
        receiver: logger
        repeat_interval: 5m
    templates:
        - test

template_files:
    test: |
        {{ define "test" }}
          asdf
        {{ end }}

Environment

• Infrastructure: Single-node monolithic GEM process on Ubuntu 22.04.
• Deployment tool: Ansible, systemd.

Additional Context

n/a

[pracucci]

Your request looks legit to me and agree we should improve the output.

Here is the fixed output I would prefer to see: alertmanager_config: | [...] template_files: test: | [...]

I think my main concern on this proposal is that it doesn't look easier to get the output and store it to files or make it easier to "pipe" to mimirtool alertmanager load. I'm wondering if, other than doing what you're proposing, we could also add a --output-dir CLI flag to allow to store the config to multiple files in the specified directory (1 file for the alertmanager, and 1 file for each template).

WDYT?

[adeverteuil]

Ah I see, mimirtool alertmanager load takes separate files for the Alertmanager config and the template files. Having an optional --output-dir parameter would be nice then, but not as important as getting valid YAML.

For comparison and context, the output of mimirtool rules print is not directly usable as an input for mimirtool rules load. I already have a habit of wrapping mimirtool calls in shell scripts to get the parameters right consistently and to do a bit of processing on the input and output.

[pracucci]

I already have a habit of wrapping mimirtool calls in shell scripts to get the parameters right consistently and to do a bit of processing on the input and output.

Just to get the big picture, how would process from the shell the output you proposed? Asking because IMO that should be part of the documentation when we'll change the output format.

[MaxDiOrio]

The rules templating is clunky at best. As @adeverteuil mentioned, you can't use the output mimirtool alertmanager get without modifications. It's not intuitive in the least!

The template file must be named exactly as you want it to appear in the contact point. If, for example, you name the file template-default.yaml, it will show up in the UI as "template-default.yaml". Not what is expected. You also can't put your templates into a directory and reference that directory. It generate an error about not having the path in the file name.

The documentation here doesn't mention the template files at all. But the documentation here does.

[pracucci]

Thanks @MaxDiOrio for your feedback. I definitely agree with you. Willing to help improving it?

[dimitarvdimitrov]

https://github.com/grafana/mimir/pull/6760 was just merged which adds --output-dir to mimirtool alertmanager get. Is there still a strong need to output a single easy-to-parse string or can we close this issue?