pixie icon indicating copy to clipboard operation
pixie copied to clipboard
verified publisher icon pixie-io

Helm uninstall leaves px-operator namespace in (stuck) terminating state

Open ddelnano opened this issue 1 year ago • 1 comments

Describe the bug The helm install process followed by a helm uninstall does not fully clean up all pixie resources in the v0.1.7 operator release. The OLM project added a csv-cleanup finalizer in v0.27.0 that causes the px-operator to get stuck in a terminating state.

To Reproduce Steps to reproduce the behavior:

  1. Install pixie via the helm install instructions
  2. helm -n pl uninstall pixie
  3. Notice that the px-operator is stuck in a terminating state due to the operators.coreos.com/csv-cleanup finalizer

Expected behavior Helm uninstall cleans up the olm, px-operator and pl namespaces.

Logs I don't have the logs handy, but I will attach them to this issue when I trigger the problem again.

ddelnano avatar May 24 '24 17:05 ddelnano

As a short term workaround, if OLM is installed prior to using the Pixie helm chart, the issue does not occur. This is because OLM won't be a part of the helm installation, which allows the px-operator finalizers defined within OLM to run successfully on helm uninstall.

I'm evaluating options for addressing this finalizer issue in the meantime.

ddelnano avatar May 28 '24 21:05 ddelnano

Can confirm I am experiencing the same issue. Have also found that installing via helm, and when uninstalling, running: px delete against the specific cluster before helm uninstall has also worked without finalizers causing operator namespace to get stuck. But yes not ideal, preferable for helm to manage everything

danlhennessy avatar Oct 30 '24 11:10 danlhennessy

I have a proof of concept on this branch that implements a potential fix.

That change adds an additional k8s Job that will delete the olm operator namespace (px-operator by default) on the helm pre-delete hook. This allows olm to stay running before helm starts deleting resources and ensures the csv-finalizer has time to run.

I'll be working to get a PR open for this once I get feedback on this approach and test out an operator rc build.

ddelnano avatar Dec 10 '24 05:12 ddelnano

#2059 is nearing completion and I hope to have an operator release out with it in the next week.

ddelnano avatar Dec 17 '24 22:12 ddelnano

#2059 is nearing completion and I hope to have an operator release out with it in the next week.

Thanks @ddelnano for the fix. Looking forward to it.

alihamza-official avatar Dec 18 '24 05:12 alihamza-official

The v0.1.7 release is now available!

ddelnano avatar Dec 19 '24 18:12 ddelnano