Nginx reload to be postponed for "$n" seconds after change is applied

[pashaliyski]

We face an issue when we make more changes regarding more than one ingress controller OR k8s service manifest configuration in our env.

In documentation, there is a block: " Batch updates. When the Ingress Controller receives a number of concurrent requests from the Kubernetes API, it will pause NGINX reloads until the task queue is empty. This reduces the number of reloads to minimize the impact of batch updates and reduce the risk of OOM (Out of Memory) errors. "

but if you do just: "find . *yaml -exec kubectl apply -f {} ;", all changes come in the k8s api with latency, not at the same time. So, if we make 100 changes in 100 different ingress manifest files, the nginx controller reloads 100 times, which cause OOM for ingress controllers.

Can you postpone the reload with $n seconds (it should be configurable?). For example:

if there are changes in any ingress manifest file, wait for 5 ($n) seconds. If there are no more changes, then reload nginx server. if there are another changes, then wait for $n seconds again. This way, if you make a batch of changes - the reload will be only one.

To Reproduce Steps to reproduce the behavior:

1 Create 100+ service or ingress definitions. 2 make changes into them 3 find . -name *yaml -exec kubectl apply -f {} ;

Expected behavior It will trigger 100+ reloads and OOM eventually.

Your environment

• Version of the Ingress Controller - nginx version: nginx/1.21.3 (nginx-plus-r25)
• Version of Kubernetes - v1.19.7

Additional context I am wondering, why ingress controller is trying to reload configuration if no changes are made? If we change service definition of any service that associated with ingress, nginx reloads its configuration, regardless any changes are made into any virtual hosts. Only config version is incremented in /etc/nginx/config-version.conf.

[github-actions[bot]]

Hi @pashaliyski thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[vepatel]

Hi @pashaliyski and thanks for opening up the issue, I see that you're using quite an old version of NIC (maybe 2.0.x) from your description and since then we've made multiple improvements around batch reloading. Can you please upgrade it to the latest stable release 3.3.2(with nginx-plus-r30). Also the documentation on website always refers to the latest stable release. Also feel free to open a ticket with F5 support if you're a Plus user.

Thanks!

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 10 days with no activity.