Make lyluatex process LilyPond snippets in parallel

[lemzwerg]

While Lua itself isn't multi-threaded, it has 'coroutines', and using them it should be possible to emulate lilypond-book behaviour, I think: If a new lyluatex option parallel=<n> (or whatever) is set, lualatex won't wait for the result of LilyPond for a lilypond environment but simply spawns a LilyPond process up to n times. This would enormously speed up everything, at the cost of running lualatex at least twice.

[jperon]

The idea is very good in itself. Nevertheless, I’m not sure I’d like to implement it:

• the process is (sometimes very) slow on "first compilation" (after a change), but the cache speeds things up on subsequent runs;
• I’m not sure the gain is worth the complexity it adds to the code: lyluatex not only compiles the score, but makes several checks that may trigger re-compilation. Yielding the right way on various places, with all conditions it would add (about previous runs, protrusion, errors…) would imply complexity, and would make the code (that’s already fairly complex, despite our efforts) less readable.

[jperon]

@lemzwerg After looking further at this, it turned not to be so complicated as I feared. I took the occasion to make a switch I had thought about for quite a while: switching to MoonScript, that’s a more concise and readable lua dialect. May you please test with the parallel branch, and tell me what you think about it? After enhancing it, I think I’d release a lyluatex v2.0, as it’s quite a big change in the way lyluatex works.

[lemzwerg]

Sorry for the late reply.

I use the new versions of lyluatex.lua and lyluatex.sty in the 'parallel' branch, doing

\usepackage[parallel=true, ...]{lyluatex}

in the main document. However, it fails soon with

(lyluatex)      Compiling score tmp-ly/0d322684d38bc38452e555c4b2ef140d with LilyPond executable 'lilypond'.

! LaTeX Error: File `tmp-ly/0d322684d38bc38452e555c4b2ef140d' not found.

Am I missing something?

[jperon]

Would you please have a MWE ? On which platform do you run LilyPond / LuaLaTeX ?

[lemzwerg]

Unfortunately, I currently don't have time to produce an MWE. However, I will send you the project privately, maybe this helps.

I'm running on GNU/Linux; however, there's nothing in the project that should prevent compilation on Windows, say (I actually did this a few years ago but haven't tested it recently with the latest updates to both the LilyPond and lyluatex code).

[jperon]

Thank you for the document. I get the same problem : I’m investigating on my free time, and will tell you what I find.