Leverage worker_threads for pre-render concurrency

[Hebilicious]

Describe the feature

The pre-render concurrency implementation relies on Promise.all, and doesn't leverage multiple cpu cores. Multithreading can be achieved with node.js using worker_threads

We could re-implement the concurrency implementation using worker threads, which would fix memory issues as we could apply concurrency based on available cores.

There's a couple libraries that makes things easier here https://github.com/piscinajs/piscina or https://github.com/andywer/threads.js

Additional information

• [x] Would you be willing to help implement this feature?

[pi0]

This is a good idea! I am thinking to refactor some part of dev server worker utils to have a more consistent experience.

[SoaresMG]

Hey @pi0, do you think this is something feasible?

I could give a hand.

Currently, I have almost 3k routes to prerender and it's super slow (~1h) on k8s.