prometheus_client_php
prometheus_client_php copied to clipboard
How can I delete / remove / un-register a specific series?
We are using this lib to push a metric called task_up that has a number of labels:
# HELP chargeover_platform_task_up Whether or not this scheduled task is working correctly (1 for yes, 0 for no)
# TYPE chargeover_platform_task_up gauge
platform_task_up{task="task1",device_id="1"} 1
platform_task_up{task="task1",device_id="2"} 0
platform_task_up{task="task1",device_id="3"} 1
This is working really great, and has already proven successful in letting us closely monitor the status of these tasks.
The problem we're running into, is that occasionally a device may fail, and be replaced, and this is expected. It's going to happen.
The issue though is that e.g. if {device_id="2"} fails and is removed and replaced by {device_id="4"}, the {device_id="2"} task metric is forever "stuck" at 0, and continues to trigger alerting. That time series is essentially "cached" in Redis forever.
In an ideal world, we could remove that series from the metrics, so that series doesn't get shown on the next scrape, gets considered stale, and no alert is fired.
Is there any planned support for this? (If not, I could create a pull request if you might be able to provide any guidance on the data structure in Redis so I can find and remove the metric?).