queue-monitor and dispatcher metrics missing

[mweinelt]

Describe the bug The queue-runner should produce four metrics that measure the internal bottlenecks of the monitor and dispatcher components.

To Reproduce Steps to reproduce the behavior:

1. curl http://hydra.nixos.org:9198/metrics | grep time_spent

Expected behavior The following metrics appear:

• hydraqueuerunner_dispatcher_time_spent_running
• hydraqueuerunner_dispatcher_time_spent_waiting
• hydraqueuerunner_queue_monitor_time_spent_running
• hydraqueuerunner_queue_monitor_time_spent_waiting

Hydra Server:

Please fill out this data as well as you can, but don't worry if you can't -- just do your best.

• OS and version: NixOS 24.11 (db8f4fe18ce772a9c8f3adf321416981c8fe9371)
• Version of Hydra: 1c52c4c0ed596ea71de370562ed5af1604bd2183
• Version of Nix Hydra is built against: nix follows input 'hydra/nix'
• Version of the Nix daemon: 2.28.2

Additional context Added in efcf6815d95134050b0f280668af7655aae9ef72

[Mic92]

@Ericson2314 do you have an idea why those metrics disappeared? You have backported them as part of https://github.com/NixOS/hydra/pull/1464

[vcunat]

They've reappeared in the meantime, but I'm not sure how. Maybe mweinelt knows.

[Mic92]

Assuming we downgraded somehow hydra? Please re-open if this persists.

[Ericson2314]

Yes no idea why that would be the case.

[mweinelt]

After another restart due to an update today we are missing the metrics again.

I think we have a problem with them not reliably getting set up.

[dasJ]

Is this fixed with the new queue runner?