Helm upgrade is often duplicated, causing issues with Jobs

[mikmatko]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Helm upgrade is seemingly called twice on every change or force update. This seems to occur most of the time, but not always.

Logs from a downstream cluster fleet-agent-0 pod:

Working scenario, Helm deployment is called only once:

{"level":"info","ts":"2024-09-18T06:36:20Z","logger":"bundledeployment.HelmDeployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"7c7639d9-f1ba-49c2-8785-b6e582c166a8","commit":"239b40d88e01e2db8d80eabeb891384b25e76311","dryRun":false}
{"level":"info","ts":"2024-09-18T06:36:37Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"7c7639d9-f1ba-49c2-8785-b6e582c166a8","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:9190b78396800308e5260944df675a40cb47e4b1c5e2180b7e70be580d38608f","release":"mikko-debug/mikko-debug:86","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452"}
{"level":"info","ts":"2024-09-18T06:36:37Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"b19b7a49-3c5f-47ad-be0f-375a7ebf5d22","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452","release":"mikko-debug/mikko-debug:86","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452"}

Then a bit later after pushing Force Update through Rancher UI (same occurs on a single new commit too):

{"level":"info","ts":"2024-09-18T06:38:01Z","logger":"bundledeployment.HelmDeployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"5a20d39e-ed74-4f98-ba97-40b6b79767e9","commit":"239b40d88e01e2db8d80eabeb891384b25e76311","dryRun":false}
{"level":"info","ts":"2024-09-18T06:38:17Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"5a20d39e-ed74-4f98-ba97-40b6b79767e9","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:17Z","logger":"bundledeployment.HelmDeployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"5ae5c18b-b800-4e8b-990e-5ce2e448bb4d","commit":"239b40d88e01e2db8d80eabeb891384b25e76311","dryRun":false}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"5ae5c18b-b800-4e8b-990e-5ce2e448bb4d","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:4f69fbccb9885a091faa2d70ed56b710a70e33c7f73d381394c10b70dbdf3452","release":"mikko-debug/mikko-debug:88","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"f327af12-2651-4550-95ba-aef2a49271c4","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.UpdateStatus","msg":"Status not ready","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"f327af12-2651-4550-95ba-aef2a49271c4","error":"job.batch mikko-debug/mikko-debug-job-87 missing"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"c80e519f-b4fd-416e-911d-d689731329da","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:88","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"47736280-52da-4da1-8114-412c7d84bf46","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.UpdateStatus","msg":"Status not ready","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"47736280-52da-4da1-8114-412c7d84bf46","error":"job.batch mikko-debug/mikko-debug-job-87 missing"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"49bfc5f3-0320-4e93-876b-c141ee5f1f7e","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:88","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:34Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"62f0793c-b15a-4bd3-9c70-46c67c7e9a7d","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:35Z","logger":"bundledeployment.UpdateStatus","msg":"Status not ready","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"62f0793c-b15a-4bd3-9c70-46c67c7e9a7d","error":"job.batch mikko-debug/mikko-debug-job-87 missing"}
{"level":"info","ts":"2024-09-18T06:38:36Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"d03de5d9-5438-4cd8-940e-f61106942130","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:37Z","logger":"bundledeployment.UpdateStatus","msg":"Status not ready","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"d03de5d9-5438-4cd8-940e-f61106942130","error":"job.batch mikko-debug/mikko-debug-job-87 missing"}
{"level":"info","ts":"2024-09-18T06:38:38Z","logger":"bundledeployment.DeployBundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"97c74c52-0b6f-4023-9e40-b1c51f078e72","deploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba","release":"mikko-debug/mikko-debug:87","appliedDeploymentID":"s-cdb5595a5910c9b95765d654ab3a84d653589f6cf4aa9f314dac997310b2b:fb4e0cef7f7c41e0faa1e9ce7662ddde11afa277c52a5b342150dd9bd23841ba"}
{"level":"info","ts":"2024-09-18T06:38:39Z","logger":"bundledeployment.UpdateStatus","msg":"Status not ready","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mikko-debug-debug-debug-chart","namespace":"cluster-fleet-default-clustername-1ba05bfd28c8"},"namespace":"cluster-fleet-default-clustername-1ba05bfd28c8","name":"mikko-debug-debug-debug-chart","reconcileID":"97c74c52-0b6f-4023-9e40-b1c51f078e72","error":"job.batch mikko-debug/mikko-debug-job-87 missing"}

As can be seen from the logs, Helm upgrade is called twice. As a result, Fleet thinks that a Job object is suddenly missing:

job.batch mikko-debug/mikko-debug-job-87 missing

While Fleet did two Helm upgrade operations, it seems to still think that it had done only once, hence looking for an object from the previous Helm release. This leaves the Bundle in a modified state.

Expected Behavior

Helm upgrade is called only once per change.

Steps To Reproduce

I believe this issue can be seen with any chart, but it is more apparent if you have any Job in the chart. Doesn't seem to matter what options are provided in fleet.yaml etc.

Environment

- Architecture: x86
- Fleet Version: v0.10.2
- Cluster:
  - Provider: GKE
  - Options:
  - Kubernetes Version: v1.30.4-gke.1213000

Logs

No response

Anything else?

No response