HelmRelease uninstalled unexpectedly

[larntz]

Today I updated flux (v2.1.1 -> v2.2.2) in a cluster and one of our HelmRelease's was unexpectedly uninstalled/installed.

We tracked this down to a change in targetNamespace, and now know exactly how this behaves in v2.2.2.

There are a couple of interesting factors here which is why I am opening this issue.

1. HelmRelease manifest did not have targetNamespace set, but storageNamespace was set.
2. The output of helm ls -a -n storage-namespace showed the namespace to not be flux-system before the upgrade (I know this because we have two identical clusters).

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
generic-chart   not-storage-ns  1               2024-01-04 18:40:00.041038688 +0000 UTC deployed       0.50.19    1.50.5

I'm not sure how this chart was deployed with helm ls showing a namespace other than the release namespace, but it has been this way for ~2 years. I'm not sure if this was done with flux, the helm cli, or something else (this was before my time).

We've never had this issue when upgrading flux in the past; until version 2.2.2 (and helm controller 0.37.2).

I see the text below as part of the helm controller 0.37.0 release notes. We did not expect this uninstall behavior because we did not specify the targetNamespace in the HelmRelease manifest.

Changes to a HelmRelease .spec which require a Helm uninstall for the changes to be successfully applied are now detected. For example, a change in .spec.targetNamespace or .spec.releaseName.

I have a few questions about this and how we might prevent or detect it before deploying.

1. I am wondering if there is some way we could prevent the uninstall behavior. I didn't see anything in the HelmRelease spec that seemed like it would work, but I wanted to ask in case I'm missing something or there is another way we could prevent this in the future.
2. I'm also wondering if could point me to a code change that would have caused this. Previous versions of flux either did not detect this as a targetNamespace change or ignored it. I didn't see any warnings/errors logged about a targetNamespace mismatch or that in the future this would behave as an uninstall/install. I'd like to understand how it worked previously vs how it works now with regard to uninstall when the targetNamespace changes.
3. Lastly, is there a way to do a dry-run with a flux HelmRelease to know if it will perform uninstall or just upgrade?

Appreciate any information you can provide would be greatly appreciated. Thank you!