Does OLM operator have a cache for CSV metadata?

[phantomjinx]

Type of question

Are you asking about community best practices, how to implement a specific feature, or about general context and help around the operator-sdk? General Question

Question

Our end-to-end testing (github actions) runs on a nightly basis and for each suite (6 of them in total), the operator, operator bundle and bundle index images are compiled and pushed to the external registry interface of our testing cluster (OCP4.10 on IBM Cloud). The tests are then geared to use these images from imagestreams and catalogsources, also created by the tests. At both the start and conclusion of each test suite, actions are executed to clean-up the cluster and leave it in as vanilla a state as possible.

However, at times the tests fail and one of the issues has been lack of permissions for the operator service account. This is easily fixable by updating the RBAC config so that a slightly different CSV is included in the bundle. And this is where things go wrong ...

Despite modifying the CSV, the CSV installed in the tests is old (eg. 2022-06-07T18:37:13).

Now I have checked:

1. The bundle image being compiled for the tests has the CSV modification;
2. The bundle image as accessed by the imagestream has the CSV modification;
3. Cleared out any images in the registry by scheduling the prune cronjob;
4. Restarted the OCP registry as this, apparently, clears out the image cache.

... And still the old CSV is being installed.

Does the OLM have an internal cache of CSVs that it maintains and periodically updates? If so, how can I clear it out, eg. restarting the OLM operator?

Thx.

Environment

• operator-lifecycle-manager version: As reported in OCP -> RELEASE_VERSION: 4.10.15

• Kubernetes version information: OCP 4.10.15_1520

[benluddy]

Just a guess: any chance you're reusing an image tag for this bundle? Bundle manifests are unpacked before installation into a ConfigMap whose name is based on a hash of the bundle image reference and not unpacked again if the ConfigMap already exists. @dinhxuanvu: Wasn't there an issue for this somewhere?

If this is the case then you can work around it by either referencing the bundle image by digest instead of tag in the catalog, or updating your teardown process to delete the unpacking ConfigMaps in OLM's namespace.

[dinhxuanvu]

We have a BZ for this and might have an issue for this at some point but probably closed it. The configmap (which holds bundle content) is not gonna be updated, it is supposed to be immutable. The expectation here is new version of the operator coming with a new bundle image (with a different tag/digest). Reusing image reference is discouraged.

[phantomjinx]

Asked and answered so closing.