Remote Cluster stuck in Wait Applied status until flee-agent-0 restarted

Open twinchester-oqc opened this issue 9 months ago • 4 comments

Is there an existing issue for this?

[x] I have searched the existing issues

Current Behavior

This is a new Rancher implementation.

Rancher v2.10.2 fleet-agent v0.11.4

Within "Continuous Delivery" Git Repos can randomly get stuck in "Wait Applied" status, seemingly forever. Drilling down to Cluster we see the problem is affecting one or more Clusters. Force Update on the cluster clears the issue. Alternatively restarting fleet-agent-0 (Same as Force Update I think) in the cluster clears the issue as well.

This affects remote clusters, does not seem to affect fleet-local cluster.

Log from fleet-agent-0:

fleet-agent-0.log

Expected Behavior

Remote cluster "Wait Applied" Status to eventually change to "Active".

Steps To Reproduce

Install Rancher 2.10.2
Add remote cluster
Add Git Repo to continuous Delivery and configure to deploy to remote cluster
Intermittently see remote cluster change to Wait Applied status.

Environment

- Architecture: arm64
- Fleet Version: 0.11.4
- Cluster:
  - Provider: AKS
  - Options: 2 Nodes
  - Kubernetes Version: 1.30

Logs

Log from fleet-agent-0:

 
{"level":"info","ts":"2025-03-27T15:44:33Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"2f1feaaa-9511-472a-ba7d-dbbd70ddf4fe","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:44:35Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"8c462574-d598-4c2a-984e-97acdaf40cf9","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:44:36Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"78c937bb-4c8a-430c-8810-3705c0d6d678","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:44:53Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"12c0586d-dd88-48df-8177-dd457c661ec1","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:44:54Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"62693a75-d06f-47aa-a668-f9988d0fad66","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:44:56Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"3805e6b1-0761-418c-a11f-6812b0ead705","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Replicas: 4/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}}]}
{"level":"info","ts":"2025-03-27T15:45:01Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"fa265f92-21ec-49f9-bb17-ef79b2f7824f","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 2"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 4/5"]}}]}
{"level":"info","ts":"2025-03-27T15:45:02Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"a8eb846d-e3ce-4eac-a7ca-297e1294ded7","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 2"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 3/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 2"]}}]}
{"level":"info","ts":"2025-03-27T15:45:07Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"e9d8c8b0-9f10-4f78-aaeb-898034f85653","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Updated: 4/5"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}}]}
{"level":"info","ts":"2025-03-27T15:45:11Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"25b4e908-11ad-4ce2-9733-1c18aa7a6115","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}}]}
{"level":"info","ts":"2025-03-27T15:45:14Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"2f030650-0280-4f40-abcf-55771f5a6f26","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}}]}
{"level":"info","ts":"2025-03-27T15:45:20Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"e7cdade1-88f0-4cd0-a709-d364f499f759","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}}]}
{"level":"info","ts":"2025-03-27T15:45:27Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"547dfbce-8848-4206-93de-c69acf026d53","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}}]}
{"level":"info","ts":"2025-03-27T15:45:28Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"2224ba96-a606-4010-a11b-54dee27ae31b","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"3af985e8-a1dc-4daa-a675-59f7b0395c6e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}},{"uid":"4c6c2a1e-c6ce-4bb9-bdb0-98898e41124e","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-jp-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}},{"uid":"969098ea-b15e-46b5-86e3-390e8bda6624","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}},{"uid":"b555701e-ace8-412b-8edb-9edac04cb522","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"toshiko-uk-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Pending termination: 1"]}},{"uid":"b7fe7835-522b-4682-ae1a-19741afe787a","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-public-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}}]}
{"level":"info","ts":"2025-03-27T15:45:32Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"qcaas-charts-qcaas-v3","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"4a7a78be-599b-43af-8e6d-10c691af67a5","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}}]}
{"level":"info","ts":"2025-03-27T15:45:32Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"qcaas-charts-qcaas-v3","reconcileID":"ef059123-1f40-466d-a1aa-bd13de9b6f5a","nonModified":true,"nonReady":[{"uid":"13e11f86-1df7-4bc8-aa5d-e352b5a71385","kind":"Deployment","apiVersion":"apps/v1","namespace":"qcaas","name":"sim1-internal-server","summary":{"state":"in-progress","transitioning":true,"message":["Available: 4/5"]}}]}
{"level":"info","ts":"2025-03-27T15:46:09Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"0245bcc7-491e-468a-aaf4-af44ec9002f1","commit":"41310437c0c59b3765ea01a98c68a850c6490fde","dryRun":false}
{"level":"info","ts":"2025-03-27T15:46:15Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"0245bcc7-491e-468a-aaf4-af44ec9002f1","deploymentID":"s-be0268e6ef308d7dc72b775933a3e473bb72b5081a0cdfa7905e24999d83a:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","appliedDeploymentID":"s-ca8c3e94e1de4e1ad6cd5d8e25d715f473d5b99f3bdcd9023f25ed4d16925:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","release":"vault-operator/vault-operator:28","DeploymentID":"s-be0268e6ef308d7dc72b775933a3e473bb72b5081a0cdfa7905e24999d83a:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30"}
{"level":"info","ts":"2025-03-27T15:51:58Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"5e7c87e1-27bc-4e0d-824a-1baf9bec164f","commit":"fb7c2ada08bd73a47797b3189892a5d05d9d2310","dryRun":false}
{"level":"info","ts":"2025-03-27T15:52:05Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"5e7c87e1-27bc-4e0d-824a-1baf9bec164f","deploymentID":"s-005e13c2a2bb434687311bf684ae307e1d52d89bc1b957c2384256b0cde51:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","appliedDeploymentID":"s-be0268e6ef308d7dc72b775933a3e473bb72b5081a0cdfa7905e24999d83a:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","release":"vault-operator/vault-operator:29","DeploymentID":"s-005e13c2a2bb434687311bf684ae307e1d52d89bc1b957c2384256b0cde51:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30"}
{"level":"info","ts":"2025-03-27T16:58:03Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"2b23a7ba-2478-4fed-8064-42faa9e00d91","commit":"f7b52c4ade30a3334ea728b181cb415effd4c386","dryRun":false}
{"level":"info","ts":"2025-03-27T16:58:10Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"2b23a7ba-2478-4fed-8064-42faa9e00d91","deploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","appliedDeploymentID":"s-005e13c2a2bb434687311bf684ae307e1d52d89bc1b957c2384256b0cde51:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","release":"vault-operator/vault-operator:30","DeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30"}
{"level":"error","ts":"2025-03-27T16:59:22Z","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"558faa29-6b5d-44a8-8e73-3a1688341a08","error":"failed deploying bundle: contents.fleet.cattle.io \"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f\" not found","errorCauses":[{"error":"failed deploying bundle: contents.fleet.cattle.io \"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f\" not found"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"info","ts":"2025-03-27T16:59:22Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"89977a36-f7c0-45cd-8a66-679e1796e651","commit":"f7b52c4ade30a3334ea728b181cb415effd4c386","dryRun":false}
{"level":"info","ts":"2025-03-27T16:59:28Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"89977a36-f7c0-45cd-8a66-679e1796e651","deploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f","appliedDeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","release":"vault-operator/vault-operator:31","DeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f"}
{"level":"info","ts":"2025-03-27T16:59:29Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"2f63f99b-b803-4dd7-9e66-cfee7541cadf","commit":"f7b52c4ade30a3334ea728b181cb415effd4c386","dryRun":false}
{"level":"info","ts":"2025-03-27T16:59:36Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"2f63f99b-b803-4dd7-9e66-cfee7541cadf","deploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f","appliedDeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:a25d0c89a220a0aab79bf0c36d0fdfc4b0e56f78f953337b15f8b65eef38ee30","release":"vault-operator/vault-operator:32","DeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f"}
{"level":"info","ts":"2025-03-27T17:20:03Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"1faa93e6-4a33-40c7-af99-e5770191717e","commit":"ae427139f77f2cd7aad75741418302406e11a8a8","dryRun":false}
{"level":"info","ts":"2025-03-27T17:20:09Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"vault-operator-fleet-vault-operator","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"vault-operator-fleet-vault-operator","reconcileID":"1faa93e6-4a33-40c7-af99-e5770191717e","deploymentID":"s-2fd3df4ca8cf3de30817c26758b98c139f152afcd71503fc02b5dbc6b3851:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f","appliedDeploymentID":"s-734935ec9775c57d9a832d654ff7fc6556a334a9d96d8a49495e1452e8c0f:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f","release":"vault-operator/vault-operator:33","DeploymentID":"s-2fd3df4ca8cf3de30817c26758b98c139f152afcd71503fc02b5dbc6b3851:63a74a73c106c4ba97995fe007fdf7d6c127bd9559bac43cfa40b87bf24bee6f"}
{"level":"info","ts":"2025-03-27T18:19:53Z","logger":"drift.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"drift-reconciler","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"istio-fleet-istiod","reconcileID":"6e85717f-fd95-4eb7-a027-393071840d34","nonModified":true,"nonReady":[{"uid":"e4ff5334-cb56-498f-b41e-94f722885b24","kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v2","namespace":"istio-system","name":"istiod","summary":{"state":"pending","error":true,"message":["the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)","Resource is current"]}}]}
{"level":"info","ts":"2025-03-27T18:19:53Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"istio-fleet-istiod","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"istio-fleet-istiod","reconcileID":"f03a8a20-0651-40b5-b925-f1013dd10288","nonModified":true,"nonReady":[{"uid":"e4ff5334-cb56-498f-b41e-94f722885b24","kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v2","namespace":"istio-system","name":"istiod","summary":{"state":"pending","error":true,"message":["the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)","Resource is current"]}}]}
{"level":"info","ts":"2025-03-27T18:19:53Z","logger":"bundledeployment.update-status","msg":"Status not ready according to nonModified and nonReady","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"istio-fleet-istiod","namespace":"cluster-fleet-default-c-xssm8-c465b88e1732"},"namespace":"cluster-fleet-default-c-xssm8-c465b88e1732","name":"istio-fleet-istiod","reconcileID":"893347ab-59fe-4613-a8cb-133b41832c84","nonModified":true,"nonReady":[{"uid":"e4ff5334-cb56-498f-b41e-94f722885b24","kind":"HorizontalPodAutoscaler","apiVersion":"autoscaling/v2","namespace":"istio-system","name":"istiod","summary":{"state":"pending","error":true,"message":["the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)","Resource is current"]}}]}
W0327 18:44:04.061984       1 reflector.go:484] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: watch of *v1alpha1.BundleDeployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 83; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
W0327 19:04:27.134327       1 reflector.go:484] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: watch of *v1alpha1.BundleDeployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 11; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
W0327 19:13:15.518559       1 reflector.go:484] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: watch of *v1alpha1.BundleDeployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 39; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
W0327 19:15:56.794236       1 reflector.go:484] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: watch of *v1alpha1.BundleDeployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 3; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
W0327 19:25:30.902545       1 reflector.go:484] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: watch of *v1alpha1.BundleDeployment ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 27; INTERNAL_ERROR; received from peer") has prevented the request from succeeding

Anything else?

No response

Mar 28 '25 18:03 twinchester-oqc

Any updates on this? Running into the same issue. Deleting fleet-agent-0, then after it restarts the cluster updates.

Jun 18 '25 22:06 dtimokhin12

2.10 will be out of bug-fix maintenance soon. Is this happening with more recent versions?

Jul 09 '25 13:07 manno

Yeah, we're seeing the same behaviour for v2.11.1

Jul 09 '25 14:07 twinchester-oqc

We have a couple versions deployed in our company. We are seeing the same behaviour in 2.10.3 and 2.11.2.

Jul 09 '25 18:07 marcus-almeida