unable to manage helm chart outside of k0sctl

[cortopy]

I just upgraded a cluster using k0sctl v0.12.4 where I initiatilly deployed a helm chart like this:

extensions:
          helm:
            charts:
              - name: cilium
                chartname: cilium/cilium
                version: 1.11.0
                values: <my_values>

I'm trying to upgrade the chart to version 1.11.1 but k0sctl will always deploy the same values and the same version as provided during cluster creation.

I have observed the following:

1. if I change the chart version and values in the k0sctl file, it gets ignored. No helm upgrade is run
2. if I run helm upgrade myself with new values and version before (1) and then run the apply k0sctl config, the helm chart is downgraded to the version and config provided during installation
3. if I delete the chart config from k0sctl, it happens the same as in (2)

Not sure if it's relevant, but I'm using dynamic configuration. I've not tried to change the k0s.k0sproject.io/v1beta1/clusterconfigs resource, as I don't know the implications of doing it directly vs k0sctl apply. Is this what I should be doing?

Another possibility I considered was editing the clusterconfig and remove the cilium chart altogether, but I don't want cilium removed for obvious reasons and I don't know if that would uninstall the chart

[kke]

Currently k0sctl doesn't do anything about the dynamic config (#270) but it will soon.

Changing the config in k0sctl.yaml won't make a difference once the cluster is in dynamic mode.

The best way to experiment would be to use sudo k0s kubectl -n kube-system edit clusterconfig k0s on a controller node. This opens the dynamic config in an editor and re-applies it once you save & exit.

[cortopy]

thank you @kke that makes sense. And if I just delete the helm chart from the clusterconfig, what happens? Is the chart uninstalled or is just ignored and I can manage it myself with my usual tooling?

[kke]

My guess is that it is ignored, @jnummelin will know better.

[cortopy]

@kke @jnummelin it's been some time but I can confirm that removing the helm chart from both the static and the dynamic config, still leaves the configuration hanging somewhere.

I just spent hours debugging why I lost connectivity in my cluster and it turned out that after I rebooted the controller node, it redeployed a very old version with cilium with a config that wasn't working. I certainly wasn't expecting that

So this means I have to remember to rollback the helm cilium release after controller node reboots each time. Is there no way to remove the helm reference from k0s?

[cortopy]

I finally managed how! For anyone with the same issue, there is a CDR called Chart (charts.helm.k0sproject.io), which is undocumented. It seems that one is created for each helm chart declared in the k0sctl configuration file or dynamic configuration. Removing the chart from the declaration file does not remove the helm release, nor the k0s Chart resource. This means it's safe to remove it from the configuration without any unexpected consequences.

However, the chart will be "updated" each time a controller reboots or the cluster is updated with the configuration set in the Chart resource. It has become an orphaned chart that is still managed in the background.

In order to remove the Chart resource I have edited it and removed the finalizer first (helm.k0sproject.io/uninstall-helm-release). I don't know if this is necessary, but I didn't want the controller to uninstall the chart, I just want to manage it with my usual helm tools.

After the finalizer is gone, one can remove the Chart and that's it. The helm release is untouched, and k0kctl no longer knows about it so it won't touch it any more.