reth icon indicating copy to clipboard operation
reth copied to clipboard

perf: dynamic worker scaling based on available workers for multiproofs

Open yongkangc opened this issue 1 week ago • 2 comments
trafficstars

Implement adaptive worker pool scaling for parallel proof workers. Currently workers are statically allocated (see #19700), causing over-provisioning during low load and under-provisioning during spikes (e.g., Xen storage node bursts).

We should dynamically spawn/shutdown workers on demand based on avaliable workers, scaling between configurable min/max bounds to reduce thrashing.

This requires benchmarks and testing.

Proposal

  • worker avaliability: if its >= 85% usage scale up, once back to 70% we can scale down back to normal
  • pending multiproofs: if pending multiproof > 200 (scale up by 20%)

Additional context

yongkangc avatar Nov 13 '25 01:11 yongkangc