bug: etcd size increasing due to continuous ingress controller updates

[connectashish5]

Current Behavior

We are observing that etcd size is rapidly increasing, when we debugged it using etcd watch command we found that ingress controller is making continuous updates to keys in etcd at regular intervals inspite of there is no change to crd objects. Due to this we are running into disk space issues very frequently.

Expected Behavior

Ingress controller should not update etcd keys unless there is a change to CRD.

Error Logs

No response

Steps to Reproduce

1. Deploy ingress-controller with 1 node.
2. Create APISIX CRDs.
3. Use etcd watch command to monitor traffic and continuous updates can be seen.

Environment

• APISIX version (run apisix version): 3.1.0
• Operating system (run uname -a): Linux
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.7

[Revolyssup]

cc: @tao12345666333

[tao12345666333]

You can change the configuration item to stop ingress-controller intervals sync

https://github.com/apache/apisix-ingress-controller/blob/master/conf/config-default.yaml

[connectashish5]

Thanks @tao12345666333 , is this the property you are referring to "resync_interval". Also, for us it is set to default value of 6h, but we see the updates happening in approx every 6-7 mins.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[indrekj]

We ran out of the default 2GB size limit and that took the cluster down. We've now set up autocompaction for etcd, hopefully, that will help in the future.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.