Use work queue for reconciliation / improvement to resiliency

[shukitchan]

If there is failure writing to redis, we should consider using a work queue to retry the event later.

We should also consider other measure to improve resiliency of the ingress controller under the pressure of high volume updates.

[shukitchan]

Example - github.com/kubernetes/client-go/tree/master/examples/workqueue

[shukitchan]

We should also consider a retry mechanism in the lua script - i.e. to retry a different pod for the service.

[shukitchan]

Reference Info on controller -

1. https://leftasexercise.com/2019/07/08/understanding-kubernetes-controllers-part-i-queues-and-the-core-controller-loop/
2. https://www.containiq.com/post/kubernetes-controllers
3. https://docs.nginx.com/nginx-ingress-controller/intro/how-nginx-ingress-controller-works/
4. https://docs.bitnami.com/tutorials/a-deep-dive-into-kubernetes-controllers

[shukitchan]

This is probably not a high priority for now.