operator-sdk
operator-sdk copied to clipboard
operator-framework
How to prefetch existing CR objects in the operator startup
Type of question
Best practices
Question
What did you do?
I've developed an operator to listen for reconcile events for a particular CRD (API). Currently, during startup, the operator reconciles all existing CRs one by one. There might be scenarios where, upon the operator restarts, there could already be thousands of CRs in the cluster. I aim to prefetch all these CRs and process them before initializing the manager.
I attempted to utilize 'manager.getClient()' to fetch the APIs using that client before invoking the 'manager.Start()' method. However, this approach failed due to the error stating that the cache wasn't initialized.
What's the best practice I should adopt here to fetch all the CRs during startup before commencing the manager?
Operator type:
/language go
I aim to prefetch all these CRs and process them before initializing the manager.
Could you elaborate on why you would have to prefetch and process the CRs? What does your operator architecture look like?
Usually when the manager starts, and the informers for the particular GVK are registered, the operator starts watching all the existing CRs in the cluster it has access to. There is no explicit step required to fetch and process the CRs, if that is something which is being handled by the reconciler.
On the other hand, if you are looking to handle external cleanups (which is not done by the reconciler), you can create a separate client and perform the action. For this, you would not have to create explicit watches. An example of the same is here: https://github.com/operator-framework/catalogd/blob/493c1de0d2851582f5ba80ea5497e45be1a4c886/cmd/manager/main.go#L229, wherein a separate metaClient is created to handle cleaning up external caches.
It would be helpful to know your use case so that we can provide more inputs.
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle rotten /remove-lifecycle stale
Still waiting on info from the author, so going to close, if this is still an issue feel free to re-open.
