fleet icon indicating copy to clipboard operation
fleet copied to clipboard
verified publisher icon rancher

How to increase bundles deployment concurrency of fleet agent

Open Mohamed-Adil-Ibrahim opened this issue 1 year ago • 5 comments

  • We have 59 uSerivces/Bundles that depends on each others without a pre-defined order at the moment, and in order for the application to be up and running we need to deploy them all together and let fleet to reconcile them - This is how we manage to deploy it using other GitOps tool but we are migrating to fleet-.

  • All of the bundles goes to "Wait Applied" state however in reality fleet agent only takes the first 5 bundles that is defined alphabetically in the GitRepo path, and some of them will go to "ContainerCreating" state since they waiting for other bundles to be deployed, but fleet doesn't continue deploying the rest of the bundles and all the bundles will stuck in "Wait Applied" state.

  • My question is: How to let fleet to deploy all the bundles configured together ?

  • The log from the fleet-agent looks like the following - omitting the bundle names -: level=info msg="Deploying bundle ....." level=info msg="Deploying bundle ....." beginning wait for 11 resources with timeout of 20m0s level=info msg="Deploying bundle ....." beginning wait for 13 resources with timeout of 20m0s level=info msg="Deploying bundle ....." beginning wait for 12 resources with timeout of 20m0s level=info msg="Deploying bundle ....." beginning wait for 11 resources with timeout of 20m0s beginning wait for 12 resources with timeout of 20m0s

Mohamed-Adil-Ibrahim avatar Mar 21 '24 13:03 Mohamed-Adil-Ibrahim

Which version of fleet are you using?

manno avatar Apr 10 '24 13:04 manno

We could make the number of concurrent reconciles configurable for the agent.

manno avatar Apr 10 '24 13:04 manno

Thanks for the reply, below are the versions I'm currently using: Rancher: v2.8.1 Dashboard: v2.8.0 Helm: v2.16.8-rancher2 Machine: v0.15.0-rancher106

Fleet Image: rancher/fleet:v0.9.1 as well as I tried rancher/fleet:v0.8.0-debug-1842-4

Mohamed-Adil-Ibrahim avatar Apr 10 '24 13:04 Mohamed-Adil-Ibrahim

We are having the same issue with our deployment and saw this thread while looking for a resolution. Is there any progress for the issue or maybe a workaround to increase the number of concurrent reconciles?

scalanova avatar Apr 23 '24 09:04 scalanova

Also related: https://github.com/rancher/fleet/issues/2552

manno avatar Jun 26 '24 08:06 manno

We increased concurrency to 50 and plan to make it configurable: https://github.com/rancher/fleet/issues/2915

manno avatar Oct 23 '24 12:10 manno

Closing as a duplicate of https://github.com/rancher/fleet/issues/2915

manno avatar Oct 23 '24 13:10 manno