1lab icon indicating copy to clipboard operation
1lab copied to clipboard

Build daemon for katex

Open SquidDev opened this issue 2 years ago • 1 comments

I've been spending some time looking at what the slow bits are of a clean 1lab build:

Name Count Total Average Max
node 1,931 6m18s 0.20s 0.53s
pdflatex 258 4m16s 0.99s 1.80s
agda 1 4m04s 4m04s 4m04s
agda html 227 48.81s 0.22s 1.12s
git 383 12.58s 0.03s 0.13s
pdftocairo 258 9.54s 0.04s 0.22s

While there's some things which are always going to be slow (Agda), the obvious issue here is KaTeX. During a fresh build, we're spinning up 1.9k KaTeX processes, each of which runs for a fraction of a second!

It would be worth experimenting with creating some sort of build daemon here; just a basic JS program which reads an equation from stdin and writes the resulting SVG to stdout. Our Shake code can then maintain a pool of these, avoiding the cost of spawning a process for each job.

SquidDev avatar Jul 28 '22 20:07 SquidDev

One other thing @plt-amy suggested is kicking off the KaTex and diagram builds earlier. We can create a separate job which parses Markdown and compiles the equations/diagrams, and start that at the same time as the main Agda build.

It won't reduce single-core time, but means we can benefit from parallelism earlier on, meaning we don't end up with a build graph like this:

A graph of our Shake build. The X axis represents time and the Y axis represents number of parallel jobs. For the first four minutes of the build, only a single job (Agda) is running. After Agda is finished, all 16 cores are fully utilised for another minute, after which the build finishes.

SquidDev avatar Jul 29 '22 14:07 SquidDev