checkov icon indicating copy to clipboard operation
checkov copied to clipboard

feat(general): POC - Add fork with Pool

Open Saarett opened this issue 4 months ago • 0 comments

We originally used a Pipe per process, and each process received a fixed batch of files ahead of time. Once a process finished its batch, it wrote the results to its pipe and exited. This caused severe load imbalance: if one process happened to receive heavier files, it worked much longer than the others. On top of that, we had a bug in how we handled the pipes, which caused processes to get stuck before actually finishing.

We switched from the pipe-based approach to a Pool, which internally works like a queue: all files are pushed into a shared task queue, and each worker takes a small batch of files, processes them, and then immediately pulls more. This ensures dynamic load balancing - heavy files are naturally spread across workers, and no worker gets stuck with a disproportionately heavy batch.

With this change, moving from the single worker under the old pipe model to 12 workers using a Pool, we improved total of runtime around 5.9× speed-up in a specific test case of a big repo.

Saarett avatar Nov 19 '25 22:11 Saarett