datadog-operator icon indicating copy to clipboard operation
datadog-operator copied to clipboard

SLO burn rate

Open melchiormoulin opened this issue 1 year ago • 4 comments

Hello, since there a customressource to define SLO https://github.com/DataDog/datadog-operator/pull/807 How can we define burn rate alert based on this SLO CR ? It seems there is some internal discussions ( this comment https://github.com/DataDog/datadog-operator/pull/807#issuecomment-1781049308 )

melchiormoulin avatar Dec 20 '23 09:12 melchiormoulin

I've found an example of slo alert, but the problem is that it requires slo-hash-id in the query.

For burn rate alert the query would be something like this:

query: "burn_rate(\"slo-hash-id\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4"

It would be great to have slo burn rate alert implemented in the operator.

vlinevych avatar Dec 20 '23 16:12 vlinevych

Hello do you think there is any chance to see this feature please @fanny-jiang @ddjamesfrullo ? :)

melchiormoulin avatar Mar 11 '24 10:03 melchiormoulin

There is possibility to create CRD only for burn rate and pass those values as attributes instead of hard-coded string query: "burn_rate(\"slo-hash-id\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4" value, and this gives us referencing parent SLO into this alert more natively, k8s operator does provide such functionality I will try work on that, in my free time.

jurabek avatar Mar 25 '24 13:03 jurabek

Hey do you think it can be implemented ? Maybe @jurabek ? It would be super nice :)

melchiormoulin avatar Jun 12 '24 07:06 melchiormoulin