gatus icon indicating copy to clipboard operation
gatus copied to clipboard

Send consecutive alerts if endpoint remains unavailable

Open mxcd opened this issue 1 year ago • 7 comments

Describe the feature request

Currently, when a service becomes unavailable an alert is being triggered once the failure threshold is reached.
If the endpoint remains unavailable, no further notification is being sent out. Keeping everyone informed about a service (still) being down, it would be nice to be able to configure consecutive emails for unhealthy endpoints after a customizable interval.

Why do you personally want this feature to be implemented?

No response

How long have you been using this project?

1 year

Additional information

No response

mxcd avatar Oct 23 '23 10:10 mxcd

Maybe adding a bool flag named consecutive under alerts, that will allow sending notification on every failure-threshold's multiple.

alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true
        failure-threshold: 2
        consecutive: true

If this works, I can implement it.

bugrakocabay avatar Oct 29 '23 06:10 bugrakocabay

Hi @bugrakocabay,
although your interpretation of consecutive in terms of wording is probably more precise, it is not exactly what I intended with this feature request.
I would imagine it to be quite annoying to be notified in the interval of the failure threshold.
Example:

interval: 60s
failure-threshold: 3

This would result in a new notification every 3 minutes.
If I would want a notification every day (as a gentle reminder to fix something that is broken) I would not be able to configure this while keeping acceptable initial triggering times.
Any thoughts on this?

mxcd avatar Oct 30 '23 18:10 mxcd

Hi @mxcd , so what you have in mind is, having the flexibility to have different time intervals for the initial alert and subsequent reminders. For instance, you want the first alert to trigger after 3 failures (in our example, that would be after 3 minutes). But for the consecutive reminders, you want it less frequent, say, once every day instead of every 3 minutes.

If so, I think having this kind of configuration would solve it;

alerts:
      - type: discord
        description: "healthcheck failed"
        send-on-resolved: true
        failure-threshold: 3
        reminder-interval: 1d

bugrakocabay avatar Oct 31 '23 05:10 bugrakocabay

Hi @bugrakocabay,

that's exactly what I had in mind. Although I also see a use-case for your consecutive: true logic in case of super-critical systems or applications.

I would opt for having both options, unless @TwiN would veto due to the configuration becoming too cluttered by offering both.

If keeping it lean is desired, I'd suggest just implementing the reminder-interval due to it's higher flexibility since it can also achieve the "3 minute consecutive reminders"

mxcd avatar Oct 31 '23 10:10 mxcd

Hi @TwiN

What is your opinion about this? How should we proceed?

bugrakocabay avatar Nov 01 '23 16:11 bugrakocabay

This is exactly what I am missing in this tool. Would love to see it in action.

AreebSiddiqui avatar Nov 19 '23 00:11 AreebSiddiqui

Related issue: https://github.com/TwiN/gatus/issues/379

amai2012 avatar Nov 21 '23 19:11 amai2012

Yep - this is a duplicate of #379. Please 👍 the aforementioned issue if you'd like to see this added.

TwiN avatar Mar 09 '24 17:03 TwiN