manager container uses latest tag resulting in possible CRD mismatch

[rich0]

I am running v0.8.0 of csi-addons and began getting crash loops with error messages referencing a missing VolumeGroupReplication CRD (and other related ones). Looking at the source these CRDs are present in the development branch but not the v0.8 tag.

Looking at setup-controller.yaml there is a :latest tag on the manager container. Changing this to v0.8.0 appears to resolve this issue, and if the manager is going to use the bundled CRDs it should probably be version tagged to ensure the two are consistent.

[reefland]

Are these the errors you saw? Using v0.8.0 and getting:

2024-06-22T15:43:56.272Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "VolumeGroupReplicationContent.replication.storage.openshift.io", "error": "no matches for kind \"VolumeGroupReplicationContent\" in version \"replication.storage.openshift.io/v1alpha1\""}

2024-06-22T15:46:10.422Z ERROR controller-runtime.source.EventHandler if kind is a CRD, it should be installed before calling Start {"kind": "VolumeGroupReplication.replication.storage.openshift.io", "error": "no matches for kind \"VolumeGroupReplication\" in version \"replication.storage.openshift.io/v1alpha1\""}

Deployed using:

---
# yaml-language-server: $schema=https://json.schemastore.org/kustomization
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.8.0/deploy/controller/crds.yaml
  - https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.8.0/deploy/controller/rbac.yaml
  - https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.8.0/deploy/controller/csi-addons-config.yaml
  - https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.8.0/deploy/controller/setup-controller.yaml

[reefland]

@rich0 - as you suggest doing an override of :latest with correct version to my Kustomization solves the crash looping.

images:
  - name:  quay.io/csiaddons/k8s-controller
    newTag: v0.8.0

[Madhu-1]

@rich0 if we are using the latest image in the yamls in the released version (this is wrong) we will take care of this in the next release to ensure that we will have only released tags for the images. @Rakshith-R can you please check this one?

[travisn]

How soon can we get a new release to fix this? Seems like all users of v0.8.0 are currently broken unless a manual workaround is applied.

[Madhu-1]

@travisn this looks like a doc problem on how to pull the yamls, I will update the doc in sometime

[travisn]

@travisn this looks like a doc problem on how to pull the yamls, I will update the doc in sometime

Thanks, I see now it was a doc problem and is fixed in the Rook repo with https://github.com/rook/rook/pull/14408, but it still seems a bug that the download link has different content than the tagged yaml.

[Madhu-1]

@travisn this looks like a doc problem on how to pull the yamls, I will update the doc in sometime

Thanks, I see now it was a doc problem and is fixed in the Rook repo with rook/rook#14408, but it still seems a bug that the download link has different content than the tagged yaml.

Yes this need some extra steps to commit changes after tagging, will revisit this later to fix it.

[Madhu-1]

we can use something like https://github.com/stefanzweifel/git-auto-commit-action to push changes to the tag during the release phase from CI. @nixpanic @Rakshith-R any thoughts?

[Rakshith-R]

we can use something like https://github.com/stefanzweifel/git-auto-commit-action to push changes to the tag during the release phase from CI. @nixpanic @Rakshith-R any thoughts?

Or should we just add install instructions in release notes(which have release download urls) and point to that from all other places to install released manifests ? (We can edit release notes and I think this is the simplest approach)

kubectl create -f https://github.com/csi-addons/kubernetes-csi-addons/releases/download/v0.8.0/crds.yaml
kubectl create -f https://github.com/csi-addons/kubernetes-csi-addons/releases/download/v0.8.0/rbac.yaml
kubectl create -f https://github.com/csi-addons/kubernetes-csi-addons/releases/download/v0.8.0/setup-controller.yaml

[nixpanic]

we can use something like https://github.com/stefanzweifel/git-auto-commit-action to push changes to the tag during the release phase from CI. @nixpanic @Rakshith-R any thoughts?

I do not think that is needed. The tagged v0.8.0 release uses :v0.8.0 in the setup-controller.yaml file.

Users that want to install from the main branch, should get the :latest version.

Am I missing something?

[Madhu-1]

https://github.com/csi-addons/kubernetes-csi-addons/blob/3ae667240c74e57dc476424cf50e3bf18328e05c/deploy/controller/setup-controller.yaml#L71 is pointing to the latest tag instead of 0.8.0 as most of the users use tagged branch to pull the yamls not from the download artifacts.

[nixpanic]

Users should be encouraged to download/fetch a tag. This repository does not even have branches for releases?

[nixpanic]

Ah, maybe I understand now.

1. the artifacts from the release page have the right :v0.8.0 version
2. when fetching the v0.8.0 tag from git, the version is set to :latest

In order to fix that, the release job should indeed push the generated artifacts into git. Overwriting (force re-push) the initially created v0.8.0 tag is maybe not nice, but it would result in the expected contents.

The CI job should probably have a force-push-to-tag after creating the manifests here (or maybe as one of the 1st steps):

https://github.com/csi-addons/kubernetes-csi-addons/blob/542ecb8dd681fcb3e58e7229b1164d787c64e457/.github/workflows/tag-release.yaml#L134-L136

[nixpanic]

For the v0.9.0 release I created a commit with the (hopefully) right version in the files, and built the release artifacts after that. When someone checks out the v0.9.0 tag, they should have the :v0.9.0 container-image, and not :latest. This was a manual step, hopefully we can find a clean way to do this in the future.

[mlow]

Git tag v0.9.0 is using the correct image tag but v0.9.1 is not (yet)

[Rakshith-R]

Git tag v0.9.0 is using the correct image tag but v0.9.1 is not (yet)

Please refer to the latest installation steps. https://github.com/csi-addons/kubernetes-csi-addons/blob/v0.9.1/docs/deploy-controller.md#installation-for-versioned-deployments

Users should refer to the documented installation steps.