Rely on API Priority and Fairness (APF) instead of client-side rate limiting

[Sijoma]

When relying on controller-runtime defaults, it's difficult to spot whether a controller is rate-limited on the client-side. This can have a negative impact on controller performance as the reconcile loop will be slowed down.

The recommendation from slack discussions seems to be to always:

1. set QPS to -1 everywhere and rely on APF
2. use the cache wherever possible

One of the concerns was that this might not be safe default when users are relying on older Kubernetes versions. I'm not sure how those older Kubernetes versions would work with an updated controller-runtime in any case? Isn't the client too new then?

Discussion links:

lavalamp: even if it works the right way you should still turn it off and rely on APF

Link to sig-apimachinery recommendation: https://kubernetes.slack.com/archives/C0EG7JC6T/p1680796612287719?thread_ts=1680791299.631439&cid=C0EG7JC6T

Controller Runtime discussion - https://kubernetes.slack.com/archives/C02MRBMN00Z/p1724224928349419