soupault icon indicating copy to clipboard operation
soupault copied to clipboard

Multithread page generation

Open linsyking opened this issue 7 months ago • 1 comments

Is it possible to generate pages by using multiple threads to speed up?

linsyking avatar Nov 19 '23 23:11 linsyking

Not yet, sadly. The main problem is that Lua-ML, the Lua interpreter that soupault uses, is not thread-safe, so it will need a global lock and whenever plugins are involved, worker threads will spend a lot of time waiting for their chance to run Lua code.

But maybe my idea is wrong and there are substantial gains from parallelizing the work that can be parallelized (such as loading page source files into element trees). I'll need to experiment with it.

dmbaturin avatar Jan 12 '24 18:01 dmbaturin