alertmanager feat: incident.io Notifier

Closes https://github.com/prometheus/alertmanager/issues/4367

Adds the technical implementation, and tests, for the incident.io notifier
Configured through the following config:

receivers:
  - name: 'incidentio-notifications'
    incidentio_configs:
      - url: '$alert_source_url'
        alert_source_token: '$alert_source_token'

Apr 27 '25 22:04 rorymalcolm

Looks good, a couple comments. It also needs docs here and here 👍

May 03 '25 10:05 grobinson-grafana

Looks good, a couple comments. It also needs docs here and here 👍

All done - I think? 🙏

May 07 '25 07:05 rorymalcolm

You have some lint failures in notify/incidentio/incidentio_test.go

May 08 '25 16:05 grobinson-grafana

You still have some failing tests I'm afraid https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372

--- FAIL: TestIncidentIORetry (0.01s)
    incidentio_test.go:48: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:48
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORetry
--- FAIL: TestIncidentIORedactedURL (0.01s)
    incidentio_test.go:69: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:69
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORedactedURL
--- FAIL: TestIncidentIOURLFromFile (0.01s)
    incidentio_test.go:91: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:91
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIOURLFromFile
--- FAIL: TestIncidentIONotify (0.01s)
    incidentio_test.go:[143](https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372#step:6:144): 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:143
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIONotify
--- FAIL: TestIncidentIORetryScenarios (0.03s)
    --- FAIL: TestIncidentIORetryScenarios/success_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/success_response
    --- FAIL: TestIncidentIORetryScenarios/rate_limit_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/rate_limit_response
    --- FAIL: TestIncidentIORetryScenarios/server_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/server_error_response
    --- FAIL: TestIncidentIORetryScenarios/client_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/client_error_response

May 10 '25 07:05 grobinson-grafana

You still have some failing tests I'm afraid https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372

--- FAIL: TestIncidentIORetry (0.01s)
    incidentio_test.go:48: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:48
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORetry
--- FAIL: TestIncidentIORedactedURL (0.01s)
    incidentio_test.go:69: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:69
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORedactedURL
--- FAIL: TestIncidentIOURLFromFile (0.01s)
    incidentio_test.go:91: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:91
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIOURLFromFile
--- FAIL: TestIncidentIONotify (0.01s)
    incidentio_test.go:[143](https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372#step:6:144): 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:143
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIONotify
--- FAIL: TestIncidentIORetryScenarios (0.03s)
    --- FAIL: TestIncidentIORetryScenarios/success_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/success_response
    --- FAIL: TestIncidentIORetryScenarios/rate_limit_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/rate_limit_response
    --- FAIL: TestIncidentIORetryScenarios/server_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/server_error_response
    --- FAIL: TestIncidentIORetryScenarios/client_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/client_error_response

Ah - apols; fixed!

May 12 '25 13:05 rorymalcolm

Hey @grobinson-grafana - just confirming this is good to go?

May 23 '25 09:05 rorymalcolm

I haven't had time to review the latest changes, but I hope to be able to do that over the weekend. If the docs are looking good and code + tests still look good I hope to merge this over the weekend.

May 23 '25 09:05 grobinson-grafana

Hello @grobinson-grafana - any chance to get this merged soon? Looking forward to make use of it. Thanks in advance!

Jun 10 '25 10:06 axdotl

Any chance to get this merge soon? It will be a great improvement over webhook.

Jun 20 '25 15:06 ankitdh7

Just heads up, we have an issue with Circle CI being broken for Alertmanager and are figuring out with other Prometheus contributors how to fix it.

Jul 08 '25 11:07 grobinson-grafana

Hi! Our CI is fixed, so will do a final review and then merge this. Thanks for your patience, I know this has been open since April.

Jul 22 '25 18:07 grobinson-grafana

Hey @grobinson-grafana - I believe this should be ready to go at this point!

Aug 06 '25 20:08 isaacseymour

👋 Hey @grobinson-grafana, would love to get this merged if you have time to review! 🙏

Aug 27 '25 16:08 isaacseymour

Hey folks, Just stumbled upon this - we really-really need this. Recently migrated to incident.io on-call and missing some very useful features due to being limited to webhook usage.

Sep 09 '25 08:09 paulyehorov

I'm not sure I follow what's going on - but I'm not able to add commits to your pull request directly despite having Maintainers are allowed to edit this pull request. enabled.

To speed things up, I've made the changes myself (as some of them are a bit pedantic e.g. ending with dots (.) on comments).

I have attached the diff of both commits I've made - I'd appreciate it if you could incorporate them and let me know your thoughts.

linter-fix.diff.txt patch.diff.txt

Most of the changes are purely cosmetic but do let me know if you have any questions on them - if you can please incorporate them I'll be happy to merge this.

Sep 22 '25 14:09 gotjosh

That is indeed mysterious: it seems like Github allows you to merge main in, but not push changes.

I've applied your patches in c22bd50. Thank you so much @gotjosh!

Sep 22 '25 15:09 isaacseymour

That is indeed mysterious: it seems like Github allows you to merge main in, but not push changes.

It does allow me to do anything through the UI. I'm not sure why is not allowing me via other means

Anyways, I've double checked @grobinson-grafana's comments and they all seem to be addressed now so this LGTM.

Sep 22 '25 16:09 gotjosh

As for the release, I'm aiming to set some time aside next week to review a few PRs, get them in and then do the release with @grobinson-grafana.

Sep 22 '25 16:09 gotjosh

Thank you very much for your contribution @isaacseymour and @rorymalcolm ❤️

Sep 22 '25 16:09 gotjosh