HigherOrderCO
Use crossbeam for scheduling work
There's no reason you'd have to write a full scheduler yourself. Crossbeam can do at least some of the work for you.
It has a work stealing deque and message passing essentials.
If you want something higher level than that, you can write this using the Actix framework. Doesn't the Actor model fit here?
Crossbeam and such (incl. Actix) are nice, very powerful "frameworks" but are very heavyweight and their performance is also not the top for "light" computations like HVM does. So I'm pretty sure this wouldn't be the most performant solution - at least compared to Weave.
To be fair, I think all we need to have a proper task-stealing scheduler is a fast thread-safe queue in C. We just push and pop must-be-normalized locations, and it should work fine, I believe.
@VictorTaelin that's 99% true with the 1% caveat :wink:. Namely a shared queue doesn't scale linearly with number of processors (cores, GPUs, FPGAs, etc.).
You can see it in Weave which implements both a work-stealing scheduler (which is not built around one "main" queue) as well as a huge queue (which one can choose instead of the default work stealing scheduler if one needs some time critical guarantees - typical for FPS games - at the expense of sublinear scalability which hurts a lot - small tens of percent with 32 processors and many tens of percent at 100+ cores).
We now have a greatly improved Rust-based parallel runtime, and we do use crossbeam... although just some minor parts of it! I wonder if we could improve our visit stack or redex bag replacing it by crossbeam's, but I'm not sure if it would really be an improvement here. Perhaps. Anyway, closing this issue to keep things organized. Feel free to open again if needed.