webgraph-rs icon indicating copy to clipboard operation
webgraph-rs copied to clipboard

par_sort_graph: Simplify using Rayon

Open progval opened this issue 3 months ago • 6 comments

progval avatar Oct 09 '25 12:10 progval

Well, the whole idea was to avoid Rayon as the bridge is horribly slow and prone to contention (that why we wrote ParFoldMap—performance with the bridge was abysmal).

vigna avatar Oct 09 '25 12:10 vigna

But we can try some benchmarks.

vigna avatar Oct 09 '25 12:10 vigna

But here we're bridging an iterator with only a small number of elements, and the current code spawns a thread for each of these elements.

progval avatar Oct 09 '25 12:10 progval

Yeah, that's the idea—we know the parallelism we want, so we do it. And no Rayon overhead.

vigna avatar Oct 09 '25 12:10 vigna

But Rayon's overhead is negligible on an iterator with less than a thousand items when processing each item takes minutes.

progval avatar Oct 09 '25 12:10 progval

This will be worked out in the current "de-parameterization" of the parallel code.

vigna avatar Nov 14 '25 19:11 vigna