fix: Too many requests to `GET /v1/servers/{id}`

[apricote]

TL;DR

In some situations hcloud-cloud-controller-manager starts to spam GET /v1/servers/{id} for a subset of the nodes in the cluster every few seconds.

Expected behavior

I would expect hccm to only send a single request per --node-status-update-frequency (defaults to 5 minutes).

Observed behavior

We get reports from customers that exhaust the rate limit and we can see that hcloud-cloud-controller-manager sends > 1 req/s for GET /v1/servers/{id}. The default rate limit is 1 request per second.

A restart of the pod fixes the behaviour.

Minimal working example

We are not sure how to reproduce this yet.

Log output

No response

Additional information

Looking at some request logs, this seems to affect even very old versions of HCCM (>2 years).