DIFF does not seem to work in case of CRD version upgrade

[Martin-Weiss]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

We deploy open-telemetry and kubewarden via fleet. In kubewarden-app bundle we get "modified" and we can not get a diff working. Assuption is that this happens because fleet deploys v1alpha1 CRD via helm chart for kubewarden-controller but kubernetes switches the object to v1beta1 - so the diff does not match properly.

See https://github.com/Martin-Weiss/rancher-fleet/blob/main/kubewarden/app/fleet.yaml#L12 - this diff does not work.

Expected Behavior

The diff should work regardless of CRD version changes

Steps To Reproduce

1. install opentelemetry via fleet
2. install kubewarden via fleet

Try to create a proper diff to get rid of "modified".

Environment

- Architecture: x86_64
- Fleet Version: fleet-103.1.2+up0.9.2 / 	fleet-crd-103.1.5+up0.9.5
- Cluster:
  - Kubernetes Version: RKE2 v1.28.10+rke2r1

Logs

No response

Anything else?

No response

[Martin-Weiss]

See https://github.com/Martin-Weiss/rancher-fleet/tree/main/open-telemetry and https://github.com/kubewarden/helm-charts/blob/kubewarden-controller-2.1.0/charts/kubewarden-controller/templates/opentelemetry-collector.yaml#L2 and https://github.com/Martin-Weiss/rancher-fleet/tree/main/kubewarden and https://github.com/Martin-Weiss/rancher-fleet/blob/main/kubewarden/app/fleet.yaml#L12

[manno]

I can reproduce this in a small experiment: https://github.com/manno/fleet-experiments/blob/main/crd-versioning/crds/crd-versions.yaml

[manno]

I was missing the namespace on my comparePatches. I can't reproduce this exact error, however I see that on 0.9.5 and 0.10 two bundles are in an errorApplied state. The agent logs:

fleet-agent-5447c777d8-sxkrc fleet-agent time="2024-07-09T10:18:09Z" level=debug msg="Bundle cluster-fleet-local-local-1a3d67d0a899/kubewarden-kubewarden-app has a dependency that is not ready: list bundledeployments: no bundles matching labels fleet.cattle.io/bundle-name=kubewarden-crd,fleet.cattle.io/bundle-namespace=fleet-local in namespace fleet-local"

However, the bundle name is kubewarden-kubewarden-crd as it is not affected by the releaseName option.

After adapting the bundle names, the deployments eventually succeed.

  # Format:
  #     <GITREPO-NAME>-<BUNDLE_PATH> with all path separators replaced by "-"
  #
  # Example:
  #
  #      GitRepo name "one", Bundle path "/multi-cluster/hello-world"
  #      results in "one-multi-cluster-hello-world".

It might be easier to match with a selector.

[manno]

@Martin-Weiss can you check the release names in your dependsOn again? I can't find any problems with multiple CRD versions.

[Martin-Weiss]

@manno - thanks for checking - for some reason I can not reproduce the problem, anymore.. so it seems the issue is gone on its own.. Will close and re-open in case we see it, again.