nr1-slo-r
nr1-slo-r copied to clipboard
newrelic
Alert defined for SLOs (Budget Perspective)
Summary
As SLOs are defined we should think about them in terms of their overall budget. If the attainment objective is 99.98 ... alerting on the rate of budget consumption versus the total amount of time remaining in the time period.
e.g. - Error SLO of 99.5 ... halfway through the measurement period we are at 99.6 attainment - meaning based on a straight line rate calculation for the SLO we are not going to make our time-bound objective.
Desired Behaviour
Alerts defined for SLOs that are sophisticated enough to execute the rate based budget consumption alerting for an SLO
Possible Solution
TBD
Additional context
Use time window and rate of consumption for the alerting context ...
- 7 day SLO alert
- 30 day SLO alert
- Specific Month SLO alert
Looking around at some examples of others implementing this, it might make more sense to have alerts along the 1h - 3d timescale.
Going to start experimenting with this by adding some combination of remaining error budget %, burn rate, and time to exhaustion at current rate to SLO/R — once I get that nailed down we can tackle alert generation using those components.
Created https://github.com/newrelic/nr1-slo-r/pull/121 to calculate and display budget remaining.
Opened https://github.com/newrelic/nr1-slo-r/issues/123 with a lot more details around what this might look like - do we want to close this card and roll the work into #123?