alloy icon indicating copy to clipboard operation
alloy copied to clipboard

Fix staleness during target handover

Open thampiotr opened this issue 9 months ago • 0 comments

PR Description

As described in the required upstream prometheus PR: https://github.com/grafana/prometheus/pull/34

In certain scenarios, the injection of staleness markers at the end of a scrape loop run is undesired. For example: if a project runs scrape in a cluster where targets are sharded across a number of instances, when a new instance joins a cluster, some portion of targets will be handed over to that new instance. The previous instance should not inject staleness markers in this case or else there may be gaps in metrics.

In this PR, we:

  • update to use the prometheus with the change from above
  • change distributed targets to be able to track which targets have moved from local instance to another remote instance in a cluster
  • disable staleness markers injection for the moved targets
  • add tests to reproduce the issue, test the distributed targets and adding some test helpers

Which issue(s) this PR fixes

Fixes https://github.com/grafana/alloy/issues/249 (at least one way in which this symptom can happen)

PR Checklist

  • [ ] CHANGELOG.md updated
  • [ ] Documentation added
  • [ ] Tests updated
  • [ ] Config converters updated

thampiotr avatar Apr 29 '24 13:04 thampiotr