Iteraring over an array

[jorgegutierrez2077]

Good day, I am using this webook to convert alertmanager request into invocation to our ticketing systems. However since alertmanager groups the events, some invocations have more than one event Oversymplyfing it :

 {
  "alerts": [
      {
          "annotations": {
              "summary": "problem one",
              "description": "found a problem"
          },
          "labels": {
              "job": "XXX",
              "severity": "critical",
          },
          "status": "firing",
      },
       {
          "annotations": {
              "summary": "problem two",
              "description": "found another problem"
          },
          "labels": {
              "job": "XXX",
              "severity": "critical",
          },
          "status": "firing",
      },
      {
          "annotations": {
              "summary": "problem 3",
              "description": "found yet another problem"
          },
          "labels": {
              "job": "XXX",
              "severity": "critical",
          },
          "status": "resolved",
      } 
   ]
 }

So right now my configuration for pass arguments uses this

  "pass-arguments-to-command": [
    {
      "source": "payload",
      "name": "alerts.0.annotations.summary"
    },
    {
      "source": "payload",
      "name": "alerts.0.labels.job"
    },
    {
      "source": "payload",
      "name": "alerts.0.labels.severity"
    },
    {
      "source": "payload",
      "name": "alerts.0.status"
    }]

Which works fine to handle the first event but if there are more than one element in the list, then the rest never get processed.

As far I could read there in no way to handle the iteration and if payload contains a list of N elements, instruct the webhook to create N invocations of the execute-command, which would be the ideal scenario.

It could also work if the payload could be stored as json file ( probably something like tempfile.HASH ) and execute command is done passing the file as argument ( leaving the execute command the responsibility of parsing the json, and removing that file once it has completed)

[moorereason]

Has you tried using the pass-file-to-command option?

[jorgegutierrez2077]

Thanks, could not make work the pass-file-to-command approach, but while checking it I stumbled upon the option of sourcing the entire-payload:

    "pass-arguments-to-command": [
      {
        "source": "entire-payload",
        "name": "alerts"
      }
    ]

And thus it can be passed as argument to a python script to load it with json.loads methods and iterate it from there.

[Just-Insane]

@jorgegutierrez2077 Sorry to bother you, but I'm trying to do a similar thing with alerts from Grafana (which now uses Alertmanager internally).

Do you have any examples you can share of iterating over the JSON array in python?

[rinaldou]

I have recently run into the same problem, iterating an array that alertmanager triggered. I used the sugetion above, where "source": "entire-payload". Thats the payload I received when setting as mentioned.

{
  "alerts": [
    {
      "annotations": {
        "description": "Servidor [192.168.24.10:9100] possui somente 9.271663129266702% ou menos livre.",
        "summary": "Servidor [192.168.24.10:9100] esta com menos de 10% livre no disco"
      },
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "ef523486461c9f0e",
      "generatorURL": "http://prometheus.int/graph?g0.expr=kong_logs_free_percent+%3C%3D+11.8&g0.tab=1",
      "labels": {
        "alertname": "kong_logsDiskSpaceFree12Percent",
        "dedicated": "plin",
        "device": "/dev/dasdd1",
        "environment": "producao",
        "fstype": "ext4",
        "instance": "192.168.24.10:9100",
        "job": "node_exporter",
        "monitor": "Prometheus LNX8018",
        "mountpoint": "/opt/kong/logs",
        "server": "LNX1045",
        "service": "kong",
        "severity": "warning"
      },
      "startsAt": "2021-12-09T13:19:59.496Z",
      "status": "firing"
    },
    {
      "annotations": {
        "description": "Servidor [192.168.24.11:9100] possui somente 9.27177797991492% ou menos livre.",
        "summary": "Servidor [192.168.24.11:9100] esta com menos de 10% livre no disco"
      },
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "4daa6f7c4954a302",
      "generatorURL": "http://prometheus.int/graph?g0.expr=kong_logs_free_percent+%3C%3D+11.8&g0.tab=1",
      "labels": {
        "alertname": "kong_logsDiskSpaceFree12Percent",
        "dedicated": "plin",
        "device": "/dev/dasdd1",
        "environment": "producao",
        "fstype": "ext4",
        "instance": "192.168.24.11:9100",
        "job": "node_exporter",
        "monitor": "Prometheus LNX8018",
        "mountpoint": "/opt/kong/logs",
        "server": "LNX1046",
        "service": "kong",
        "severity": "warning"
      },
      "startsAt": "2021-12-09T13:19:59.496Z",
      "status": "firing"
    }
  ],
  "commonAnnotations": {},
  "commonLabels": {
    "alertname": "kong_logsDiskSpaceFree12Percent",
    "dedicated": "plin",
    "device": "/dev/dasdd1",
    "environment": "producao",
    "fstype": "ext4",
    "job": "node_exporter",
    "monitor": "Prometheus LNX8018",
    "mountpoint": "/opt/kong/logs",
    "service": "kong",
    "severity": "warning"
  },
  "externalURL": "http://alertmanager.int",
  "groupKey": "{}/{alertname=\"kong_logsDiskSpaceFree12Percent\"}:{alertname=\"kong_logsDiskSpaceFree12Percent\", service=\"kong\"}",
  "groupLabels": {
    "alertname": "kong_logsDiskSpaceFree12Percent",
    "service": "kong"
  },
  "receiver": "konglogspaceclean",
  "status": "firing",
  "truncatedAlerts": 0,
  "version": "4"
}

In my case, I wanted just the IP address of each server alerted. which was located in "alerts.labels.instance". So the bash script below would filter the instance, remove the unnecessary strings, and created an array with only the ip address.

#!/bin/bash
# save the entire payload in a temp file
echo $1 > /home/linux/webhook/tmp/json.txt
#filter the json, creating an array with only the ip address 
IPs=$(jq -r '{ instance: .alerts[].labels.instance }' /home/linux/webhook/tmp/json.txt | \
  sed 's/{//g' | sed 's/}//g' | sed 's/"instance"://g' | sed 's/:9100//' | sed 's/"//g')
#iterating the variable
for ip in $IPs
do
  sudo /usr/bin/ansible-playbook -i $ip, --private-key \
    /home/linux/.ssh/id_ecdsa /usr/local/share/ansible/webhook/webhook.yaml
done

Hope it helps anyone trying to webhook with alertmanager.