sensu-go icon indicating copy to clipboard operation
sensu-go copied to clipboard

Feature Request: Add acknowledgement workflow

Open asachs01 opened this issue 5 years ago • 2 comments

Expected Behavior

As a user, I'd like to have an acknowledgement workflow in Sensu Go that works like the following:

  • An alert is received via a communication handler (Slack, email, SMS, etc)
  • A link is supplied in that handler to acknowledge/silence the alert
  • The link would be prepopulated with whoever acknowledged/silenced the alert and for how long

Current Behavior

This behavior doesn't exist

Possible Solution

Ideally, this could be an extension of how contact routing worked in Sensu Enterprise. A couple of things come to mind:

  • Templates: the alerts could be configured via template and the user, silenced duration, and reason could be part of that
  • This workflow could tie in with the paging/alerting handlers like Pagerduty, Victorops, and Opsgenie and silence/ack alerts there as well

Context

Original request:

As an alert comes out to whichever destination (Slack/Email/etc), it would be good to be able to supply an acknowledgement link that can be clicked, it would auto silence the check for a configurable period (maybe default 30minutes).
It could then also lookup the handlers the event was sent to and just send a basic notification back out saying "host/checkname was acknowledged by user x".
The idea around the default silence period would be so that if it requires more than 30 minutes to resolve the user simply extends it.
Likewise if they acknowledge the issue but then never do anything about the silence entry would then expire - so there is no danger of things being ignored.

We have quite a large IT team and a feature like this just helps situations where multiple engineers might try to jump on an issue at the same time - and makes the ownership of the problem very clear.

asachs01 avatar Mar 18 '19 21:03 asachs01

Love this!

I'd bucket this one under our need for a "Webhooks API", which would facilitate two-way integrations such as you've described here (and many more).

Our assumptions on how we'd approach this up until now have always been that a Webhooks API is basically a shim in front of the Events API. We have also thought of adding support for new Webhook Sensu resource that contained a mapping of incoming event payloads/fields to Sensu Event fields/payloads (and potentially adding support for some default values). Creating a new Webhook resource would create a new Webhook API route or endpoint (e.g. /webhooks/slack).

This is still floating around on our roadmap for implementation in 2020, so stay tuned!

calebhailey avatar May 06 '20 03:05 calebhailey

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Nov 02 '20 03:11 stale[bot]