alloy
alloy copied to clipboard
Fix staleness during target handover
PR Description
As described in the required upstream prometheus PR: https://github.com/grafana/prometheus/pull/34
In certain scenarios, the injection of staleness markers at the end of a scrape loop run is undesired. For example: if a project runs scrape in a cluster where targets are sharded across a number of instances, when a new instance joins a cluster, some portion of targets will be handed over to that new instance. The previous instance should not inject staleness markers in this case or else there may be gaps in metrics.
In this PR, we:
- update to use the prometheus with the change from above
- change distributed targets to be able to track which targets have moved from local instance to another remote instance in a cluster
- disable staleness markers injection for the moved targets
- add tests to reproduce the issue, test the distributed targets and adding some test helpers
Which issue(s) this PR fixes
Fixes https://github.com/grafana/alloy/issues/249 (at least one way in which this symptom can happen)
PR Checklist
- [ ] CHANGELOG.md updated
- [ ] Documentation added
- [ ] Tests updated
- [ ] Config converters updated