Derek Su

@philippedev101 Feel free to reopen the PR and move it forward. Thanks.

@Khalid-Nowaf Feel free to reopen the ticket if you want to continue and revise the PR. Thanks.

Always failed to rebuild newly added replica ``` 2024-08-22T09:16:33.794528643Z time="2024-08-22T09:16:33Z" level=error msg="Failed to rebuild replica 10.244.244.251:10000" func="controller.(*EngineController).startRebuilding.func2" file="engine_controller.go:1960" controller=longhorn-engine engine=pvc-2064afaf-8de8-48e4-959d-188ed2dd99b3-e-0 error="proxyServer=10.244.122.244:8501 destination=10.244.122.244:10012: failed to add replica tcp://10.244.244.251:10000 for volume: rpc...

> Setting `revisionCounterDisabled` tor `true` for the engine does not work. When the volume get attached, it switches to faulted state. Sounds weird. Can you provide a support bundle again?...

cc @PhanLe1010

@emilburzo Can you provide the CPU usage over time of longhorn-manager and instance-manager pods?

> @derekbit I'll try, I don't have any monitoring setup which is so granular > > or is there a native kubernetes way of getting this I might not know...

@emilburzo Thanks for the update. Is the load still high after detaching all Longhorn volumes? Besides, can you check if the applications that are using Longhorn volumes have high CPU...

> I'll try and go backwards: uninstall everything (prometheus, grafana, longhorn) and start adding pieces one at a time (just install longhorn, setup, add a volume, etc) and see if...

@emilburzo Thanks for the update. Longhorn needs to periodically synchronize the cache with etcd, check the node and disk status and so on. We need to investigate the impact of...