Scale ratelimit with ingress pods

[dbaumgarten]

Proposed changes

Implements #4894

Add a flag (in VS-Policy and in Ingress-Annotations) to enable ratelimit-scaling. If enabled, the configured ratelimit is divided by the number of currently active ingress-pods. This way the total sum of the ratelimit stays constant, even when autoscaling changes the number of nginx-ingress pods.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• [x] I have read the CONTRIBUTING doc
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked that all unit tests pass after adding my changes
• [x] I have updated necessary documentation
• [x] I have rebased my branch onto main
• [x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[netlify[bot]]

👷 Deploy request for nginx-kubernetes-ingress pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	5323b23e285842c2ee75e9d9ace40aa23eb97d95

[dbaumgarten]

@shaun-nx What do you think? Does this look fine or are any changes required?

[brianehlert]

Just an FYI as this has been lingering for review. The NGINX Plus version of the implementation should get accurate rate limiting across all pods with state sync. That comes with the shared memory zone being configured for the rate limit and then state sharing being enabled.

[dbaumgarten]

Rebased on current master and added a commit clarifying the docs about the limitations of this feature.

[j1m-ryan]

Hi @dbaumgarten, thanks for following through, just an FYI that github actions are not working properly at the moment, that is why the tests are not running. https://www.githubstatus.com/