Generate API request to copy for SLO series deletion

[ArthurSens]

We've seen a couple of times that we've instrumented our apps incorrectly, resulting in metrics representing wrong data and affecting our error budgets.

We later fix our instrumentation, not changing the metric name or labels, just changing when they increase, but the damage is already done and we can't fix our SLO. It would be nice if Pyrra could reset the SLO for us somehow.

[ArthurSens]

Maybe calling the delete series api?

[metalmatze]

Good question! :+1: Yes, so far I recommend that folks use the delete series API of Prometheus to delete the series in the Prometheus server.

I guess I'm not too keen on adding this to Pyrra directly. It's pretty invasive and so far Pyrra is read-only when it comes to talking to Prometheus. What if something goes wrong and the wrong series is deleted, or it's too easy to do and someone deletes these by accident? Maybe we can have some way of generating the API call that would be needed to be run against a Prometheus to delete the series for a request. Even in that case, you'd probably want to adjust the timestamps manually. If added to Pyrra, it needs more thought, I think.

[ArthurSens]

Good question! 👍 Yes, so far I recommend that folks use the delete series API of Prometheus to delete the series in the Prometheus server.

I guess I'm not too keen on adding this to Pyrra directly. It's pretty invasive and so far Pyrra is read-only when it comes to talking to Prometheus. What if something goes wrong and the wrong series is deleted, or it's too easy to do and someone deletes these by accident?

Fair point! I was not thinking of something that can't be accidentally triggered. Maybe a separate endpoint that is not accessible unless you know it, or maybe a button that requires a second confirmation and a clear message that you're about to perform something undoable.

Here is how HoneyComb sends this message for example:

Maybe we can have some way of generating the API call that would be needed to be run against a Prometheus to delete the series for a request. Even in that case, you'd probably want to adjust the timestamps manually. If added to Pyrra, it needs more thought, I think.

Not everyone that is using Pyrra knows which recording rules are created by it, so generating the API call, even if we need to execute it manually, is already a good improvement ⬆️