Regression in the 5.9.8 Release

[rajatvig]

Issue Details

Post an upgrade from 5.9.4 to 5.9.8, we noticed that the metrics for running builds are not getting updated after the builds complete. This behaviour causes a change in scaling behavior as metric calculation we use sums running and scheduled builds for a queue to decide if there are enough agents running. The metric we see the same value for is buildkite_queues_running_jobs_count.

Setup

We are running unclustered agents and running the agent metrics binary to export metrics to Prometheus.