Review controller reconciliation loops to reduce the API server load and fix policy status update errors

[jvanz]

This is a spin off of the issue #645. In that issue an user was complaining that the controller is doing too many requests to the API server. Specially regarding to the configmap resource. The controller is hitting the API server many times to get the configmap because all the reconciliation loops uses that information. However, this resource is not cached in the controller. Therefore, the first mitigation proposed in that issue was enable cache for the configmap resources. In the same issue another users reported that the controller is updating the policy status many times. This is also true. Because the reconciler waits for the policy server reconciliation to properly set the policy status. And every time policy server status is "not ready", it update the policy status and requeue a request to check again. Therefore, many request updating the policy status, even with the same status value, are sent.

Both issues, the many get requests for the configmaps and the many policy status updates has a common root. All are related to the way the reconciliation loops wait for each other and get/updates many resources in the process. Therefore, we should review how the 3 reconciler work together and try to simplify them reducing the amount of time the policy status are updated and how many times we fetch data from the API server.

Acceptance criteria

• Investigate if we can simplify the reconcilers in the controller reducing the number of resources requested and policy status updates over time.
• Simplify the controller reconciliation loops where is possible.