Start using WorkerQueues

[jelmersnoeck]

We're currently handling all incoming requests the moment they occur. This means new resources, resource updates and resyncs will immediately trigger an action.

This is fine under low impact, but when we have multiple Projects and Resources configured, we end up having an unpredictable amount of syncs happening at once. This both hits our API Server (fixed by using Cache Informers) but also hits the Manifold API all at once. We should be more careful about this chatter.

For this, we'll implement a WorkerQueue. This will allow us to set up a number amount of workers (preferably configurable) which only handle n amount of syncs concurrently, where n is the number of workers configured.

This means we'll always have a steady performance pattern and can rely on the resources we configure for the controller.

To do this, we'll have several queues, one for each CRD we have. We can then start the queues and gradually pull things off the queue and process them.

An example to use some helper methods can be found in the Grafana Operator.

[jelmersnoeck]

For review reasons, I think it'd be good to split this up into multiple parts also. My suspicion is that it could end up being a big change otherwise.