Fluxcd addon upgrade issue from 1.3.5 to 2.1.0

[tanviranadive]

While upgrading Fluxcd addon version from 1.3.5 to 2.1.0 (Kubevela version being updated from 1.4.1 to 1.5.4 as well), we see that some CRDs are getting deleted and recreated every few seconds in the reconciliation loop. Vela CLI version - v1.5.4

Steps to reproduce:

Upgrade Kubevela version from 1.4.1 to 1.5.4 Upgrade addon fluxcd version from 1.3.5 to 2.1.0 Upgrade addon terraform from 1.0.9 to 1.0.13

Multicluster is not enabled in our case.

During upgrade, crds created by fluxcd are getting patched and deleted for eg. helmrepositories.source.toolkit.fluxcd.io, gitrepositories.source.toolkit.fluxcd.io As a result, the addon terraform is stuck in enabling status because helm crd is not found.

Fluxcd is deleting existing crd of fluxcd.io and tries to recreate every interval. This is happening in a loop every few seconds. Fluxcd addon application status shows running -

See below screenshot for resourcetrackers:

We tried only upgrading Kubevela version and kept fluxcd addon version same and that is working fine. We suspect there is some change in the new version on fluxcd which is recreating the crds.

Please see the vela core logs for more details:

kubevela-vela-core-logs.txt

[wangyikewxgm]

Have you updated all CRDs of KubeVela before upgrading vela-core?

[tanviranadive]

Yes, we have a dependency to update the CRD before updating vela-core. I have also attached the other versions root and v1 and v2 for the resourcetracker addon-fluxcd resourcetracker_addon_fluxcd.txt resourcetracker_addon_fluxcd_v1.txt resourcetracker_addon_fluxcd_v2.txt

[wangyikewxgm]

Yes, we have a dependency to update the CRD before updating vela-core. I have also attached the other versions root and v1 and v2 for the resourcetracker addon-fluxcd resourcetracker_addon_fluxcd.txt resourcetracker_addon_fluxcd_v1.txt resourcetracker_addon_fluxcd_v2.txt

ping @Somefive , the resourceTrackers info are here, please help to check.

[tanviranadive]

And the ResourceTracker CRD is here: resourcetrackers_crd.txt

[tanviranadive]

I can reproduce this issue using the Kind cluster. Here are the steps for reproducing the issue. Please let us know if you need more information.

Kind cluster creation: kind create cluster --name kubevela

Kubevela 1.4 install steps:

Vela-cli version: 1.4.1 (https://kubevela.io/script/install.sh | bash -s 1.4.1)

1. Install kubevela through the following command vela install
2. Verified Kubevela core installation and everything seems good.
3. Enabled fluxcd with the following command - vela addon enable fluxcd --version 1.3.5
4. Verified the fluxcd installation everything seems good.
5. Enabled terraform addon with the following command - vela addon enable terraform --version 1.0.9
6. Verified the fluxcd addon installation everything seems good.

Kuvbevela 1.5.4 upgrade steps: Vela-cli version: 1.5.4 (https://kubevela.io/script/install.sh | bash -s 1.5.4)

1. Install kubevela core 1.5.4 through the following command -vela install and prompted for the overwrite and have type yes.
2. Verified Kubevela core installation and everything seems good.
3. Enabled fluxcd with the following command - vela addon enable fluxcd --version 2.1.0
4. The fuxcd installation is not successful. The fluxcd CRDs keep deleting in a loop.

[Somefive]

I'm looking into this issue with highest priority. Maybe need several days to solve.