Support for webhooks from Prometheus, GitHub, ...

[genofire]

It would be nice if ntfy.sh could receive the json format from alertmanager: https://www.prometheus.io/docs/alerting/latest/configuration/#webhook_config

And the Annotiation shoud be the Title, generatedURL as message and labels as Tags exclude label "priority" should be used for Priority

PS: With one request multiple messages would be created ...

[binwiederhier]

Fascinating idea.

[binwiederhier]

Should be easy to implement. Not sure when I'll get to it, but if you want to give it a shot I'll review the code.

[cmeis]

Adding to this, a generic way to publish via POST with JSON body (parameters like in URL parameters and HTTP headers) would be fantastic (and shouldn't be too hard to implement - unfortunately my Go is non-existent :-/ ).

[binwiederhier]

Adding to this, a generic way to publish via POST with JSON body

I think this idea and OP's request are very much distinct. What you are asking for, @cmeis, is something like this:

curl -d '{
  "topic": "mytopic",
  "message": "This is a message",
  "title": "My title"
}' ntfy.sh

That's not a bad idea, though I'm not sure I see the benefit over the other publishing methods. Mayyybe that's easier to write code for, since you don't have to do weird header things... Not sure if it's worth it. Feel free to open another ticket for it. It is very much distinct from this ticket though.

It would be nice if ntfy.sh could receive the json format from alertmanager

I still like this idea, but I think there is no way to make this generic to support all sorts of webhooks. For instance, I'd love to support the github webhooks, but they, too, have a very specific format that I'd have to support and parse and map; and you'll get into a race of what's supported and what's not; and then in the end all you do is add integrations for other tools.

I think I'll eventually implement this at least, but I wish there was a more generic way:

curl -d '{...github json...}' ntfy.sh/mytopic/github
curl -d '{...prometheus json...}' ntfy.sh/mytopic/prometheus

[cmeis]

Yepp, reading it again this has nothing do with the original ticket - I'm outta here.

[rogeliodh]

I think it would be better to create bridges, similar to https://github.com/DRuggeri/alertmanager_gotify_bridge as separate projects from ntfy, else ntfy code will grow every month as new users will want integration with the latest™ and better™ tool

Or maybe set some requirements for built-in integrations (only for very well known products, with stable, documented and versioned API).

Or maybe with a plugin system like https://github.com/hashicorp/go-plugin ?

[binwiederhier]

@12nick12 opened #154 with essentially the same feature request. Given that now 4 people (5 including me) want this, I am promoting this feature to "hot" :-D

I still don't know how to implement this. There are a thousand ways, but I like the "plugin" idea the most, as @rogeliodh brought up. In the past, I have implemented subprocess-based plugins based on STDIN/STDOUT communication (unlike go-plugin which is grpc), which makes it trivial to do, even with bash scripts. What I don't like about that is that it's very much against the "batteries included" mentality of ntfy. I don't want to tell people "go make a plugin", ...

:thinking: :thinking: :thinking:

[12nick12]

I';m sure there's a better way, but I was thinking since each message has a body we could do some sort of template. For example the data below just use a parsing format like jq. If we could either config via GUI or yaml files that'd be awesome. I know a lot of people would like to be able to config via GUI, but I like being able to keep my conifg in git.

Title = $project, $job.id Message = This $trigger was executed with $execution.id with the status of $execution.status.

{
  "trigger": "success",
  "status": "succeeded",
  "executionId": 179,
  "execution": {
    "id": 179,
    "href": "https://rundeck.domain.com/project/git-project/execution/follow/179",
    "permalink": null,
    "status": "succeeded",
    "project": "git-project",
    "executionType": "user",
    "user": "USER",
    "date-started": {
      "unixtime": 1646766495415,
      "date": "2022-03-08T19:08:15Z"
    },
    "job": {
      "id": "13f3cc36-6830-4bcb-add7-9d56102a915c",
      "averageDuration": 185386,
      "name": "apt-update",
      "group": "",
      "project": "git-project",
      "description": "",
      "href": "https://rundeck.domain.com/api/40/job/13f3cc36-6830-4bcb-add7-9d56102a915c",
      "permalink": "https://rundeck.domain.com/project/git-project/job/show/13f3cc36-6830-4bcb-add7-9d56102a915c"
    },
    "description": "Plugin[com.batix.rundeck.plugins.AnsiblePlaybookWorflowNodeStep, nodeStep: true]",
    "argstring": null,
    "serverUUID": "ff7afb2e-56a8-4ca4-a181-bce45674d1ef"
  }
}

[binwiederhier]

@12nick12 I feel like your example is incomplete. Typically in application X (github, prometheus, rundeck, ...) you configure a target webhook address and that's it. Are you proposing we'd have a per-topic .yml file that would know how to parse incoming messages?

If so, then I am not a fan of that idea. Ideally I'd love to have a zero-config solution. Something that's either a "batteries included" approach, or infinitely configurable just form the webhook URL or via headers.

Option 1: Batteries included: We implement a per-service URL, like https://ntfy.sh/mytopic/github and https://ntfy.sh/mytopic/rundeck, and pick sensible defaults of what should be displayed. This is how Discord has implemented webhooks. The upside is that it's easy for users, the downside is that it's not configurable and likely annoying in the longrun, because people have different preferences and will just want to add more and more services.

Option 2: Configure via URL/headers: We allow templating in title and message, but only if template=on (or template=json or ...), maybe like this:

POST /mytopic/hook?template=on&title=${project},+${job.id}&message=This+${trigger}+was+executed+with+${execution.id}+with+the+status+of+${execution.status}. HTTP/1.1
Host: ntfy.sh

{
  "trigger": "success",
  "status": "succeeded",
  "executionId": 179,
  "execution": {
    "id": 179,
...

The expressions in ${...} could be parsed by gson. This kinda sounds like fun to implement.

[aTable]

I havent used go before so I went with node and made my own simple bridge that might be of interest. I know its js but at the very least, you'll be able to cherry pick the Prometheus+Alertmanager docker setup and point it to a go server so that'll simplify the dev environment setup. The incoming requests are logged if you wanted to create more complex prom stack setups but ive included a sample payload in the repo for quick reference that tells you all you need about the schema. It's a partial option 2 solution (but message is hardcoded cause jq headaches) and the alertmanager config looks like:

  webhook_configs:
      - url: http://ntfy_alertmanager_bridge:30000/ntfy_alert?topic=.labels.topic&title=.annotations.summary&priority=.labels.priority|tonumber
   

As this was a prototype built in a few hours, I don't expect to be doing any further changes on it. It'd be great if there was a way to create a markdown templated message this way ..

[prabirshrestha]

A generic way to solve this would be to support lightweight scripting.

https://www.webscript.io/ is a great example for this. You choose a url type in the lua script. Since lua is a programming language you can write any complex logic. Website has examples on how you can send emails, sms, http request and so on. Since lua language is small and the library is small it is you to the ntfy to provide.

You could provide a ntfy module or global variable to the scripting language and provide some core functions such as sms, smtp, httprequest, json, querystring, file upload, multipart, exec, ssh, database. You might want configuration where you disallow risky command such as exec which can execute arbitrary command on the server. For functions such as ssh, you might also want to have a general key/value store or environment variable concept where it can be shared, though at first it can easily be hardcoded in the script file.

Here are some potential languages you could embed.

• https://github.com/yuin/gopher-lua
• https://github.com/dop251/goja (javascript)
• https://github.com/robertkrimen/otto (javascript)
• https://github.com/d5/tengo

This could allow to replace tools such as asw lambda functions.

[benisai]

Interested in this. Make it hot again

[xenrox]

I have written a configurable bridge between Alertmanager/Prometheus and ntfy. I will probably add the ability to parse templates in the future so that the alert body and title can be customized further.

Maybe this project is of some use to other people as well.

@binwiederhier If you are still interested in adding first class support, I am willing to help out. But to be honest I think writing bridges makes more sense.

[binwiederhier]

@xenrox Very cool. Thank you. I added it here https://github.com/binwiederhier/ntfy/commit/5b8372d260435f3b1e8198ec9c6f1eb2a2ace780. It is worth noting that there is another bridge @aTable wrote here: https://github.com/aTable/ntfy_alertmanager_bridge

I would love to add it natively, but I think it's a little too out of scope, and I don't want to suddenly integrate all the tools.

[xenrox]

@xenrox Very cool. Thank you. I added it here 5b8372d. It is worth noting that there is another bridge @aTable wrote here: https://github.com/aTable/ntfy_alertmanager_bridge

Yes I saw that one but had a few issues with it that kept me from using it/contributing, so I decided to create my own bridge:

• No license, so not really useable
• The configurability was not sufficient enough for my usecase
• I prefer Go to JS

I would love to add it natively, but I think it's a little too out of scope, and I don't want to suddenly integrate all the tools.

Yeah I agree there, will continue to improve my bridge then.

[ksurl]

Webhook support is my last use case to ditch discord for notifications. I'm using it to monitor events in gitea and github actions

[genofire]

@xenrox thanks for you image, i have write an helm-chart for it: https://dev.sum7.eu/genofire/k8s/src/branch/main/cluster/charts/ntfy-alertmanager

I got few issues:

• like to have an tag for resolved alerts
• actions in notification
  • open alertmanager's url:
    • as Action: https://docs.ntfy.sh/publish/#open-websiteapp
    • as Click: https://docs.ntfy.sh/publish/#click-action
  • or maybe silence it with: https://docs.ntfy.sh/publish/#send-http-request
• strange configfile format, i have to remap it from helm's values yaml file ...) - i prefer https://github.com/knadh/koanf there a multiple parser and provider, from yaml, toml to enviroment variables ...

---

i ask myself, if it is possible to edit old notifications (to be resolved, instatt of create new notification)

[xenrox]

@genofire Currently ntfy does not support editing old notifications, so for now sending the resolved alert as its own notification is probably the best way.

I have added a few tickets for your issues. If you have further issues, you can submit them or comment on existing ones by sending a plaintext email. Furthermore you can just write me directly or to the mailing list, so we don't spam this issue here too much.

[binwiederhier]

I think the discussion in this ticket is quite interesting, but right now I don't see a good way forward with native webhook integration. It seems too far away from the push notification and pubsub core idea of ntfy. And as others have said, bridges do seem to make more sense.

I'm closing this, but I'm happy to keep discussing. Thank you all for contributing to the discussion!