sphinx-doc
Use Latexmk in a more efficient way for `'xelatex'` as `latex_engine`
Subject: improve (slightly) efficiency of xelatex builds
- Refactoring
We have said for a long time in our docs:
Also, if latexmk is at version 4.52b or higher (January 2017) LATEXMKOPTS="-xelatex" speeds up PDF builds via XeLateX in case of numerous graphics inclusions.
After some complex chase (as I could not find a Change log with release dates at Latexmk home going far back enough), it appears that from this Changes files without dates and CTAN announcements that both -xelatex and -lualatex options have been available since at least 2013, but that there was important changes at 4.52 of January 2017 to which the above tip was actually referring.
Starting from this date the -xelatex option triggered faster builds by Latexmk because it produces the PDF only once at very end of the multiple LaTeX passes. In the intermediate passes it produces .xdv files which is a bit faster process than producing PDF file.
What my PR does is to modernize our Makefile to use that -xelatex option (and -lualatex). Currently I can only test it on TeXLive 2019 LaTeX distribution not on earlier ones (and of course on current TL2024). I thus think that the change will not impact Sphinx users, even with very obsolete LaTeX installations. Due to this I have added it to 8.x milestone but hesitate about asking for inclusion in 8.0.0 (ping @AA-Turner). I will be able to test on antequated systems only in a few weeks, and depending on my schedule during transfers perhaps not until September.
Only the Makefile is changed, the latexmkrc Perl configuration appeared to require no change.
One should not expect miracles though in terms of performance improvement.
Commits will be added to update docs after opening the PR.
Relates
- https://github.com/python/docsbuild-scripts/issues/169
- #4159
Testing make -C _build/latex all-pdf after make latex with a build of Sphinx own docs configured to use xelatex on a TL2019 is a bit disappointing.
note that these builds are on a very old laptop and everyone else is going to observe faster build times.
With this PR (xeatex 2019):
real 1m23.364s
user 1m20.824s
sys 0m1.979s
Without (of course cleaning all auxiliaries from latex build first) (xelatex 2019)
real 1m22.613s
user 1m28.739s
sys 0m1.992s
WIth TeXLive 2024:
With this PR (xelatex 2024):
real 2m13.976s
user 2m9.313s
sys 0m3.925s
Without this PR (xelatex 2024):
real 2m15.046s
user 2m18.895s
sys 0m4.023s
Each build uses 3 LaTeX passes.
Observe the way LaTeX is slower nowadays...
I also observed some warnings which look inocuous
xdvipdfmx:warning: Creating ToUnicode CMap failed for "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Regular-400.otf"
xdvipdfmx:warning: Failed to load ToUnicode CMap for font "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Regular-400.otf"
xdvipdfmx:warning: Creating ToUnicode CMap failed for "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Solid-900.otf"
xdvipdfmx:warning: Failed to load ToUnicode CMap for font "/usr/local/texlive/2024/texmf-dist/fonts/opentype/public/fontawesome5/FontAwesome5Free-Solid-900.otf"
(anyhow when xelatex produces a PDF it does it via an intermediate .xdv which is then removed after xdvipdfmx, but perhaps then the warnings for some reason do not show). Anyway the return status was 0.
Now, it is expected that there will be gains when the PDF uses a lot of graphics inclusions, which is not the case with our own docs.
Not having experimented with that, I see no strong case to merge this PR in time for 8.0.0 and risk some downstream problems. If volunteers have time to try out this branch on their project and report...
If globally the gains are non-existent as above timings indicate, maybe better not to change anything at all.
In the meantime the advice seems to be to stick with some TeXLive 2019 based LaTeX rather than rush to modern one...
edit: I had been wondering if my TL2019 binary xetex was perhaps exceptionally fast due to having been built from source with compiler optimizing flags, but I have now used it with the TL2024 LaTeX macro layer and there is no significant difference with what the tL2024 binary does.
(xelatex with this PR, binary from 2019, LaTeX macro layer from 2024)
This is XeTeX, Version 3.14159265-2.6-0.999991 (TeX Live 2019) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode
(./sphinx.tex
LaTeX2e <2024-06-01> patch level 2
L3 programming layer <2024-05-27>
(./sphinxmanual.cls
....3 passes plus xdvipdfmx at the end...
real 2m16.333s
user 2m12.578s
sys 0m3.061s
I added the do not merge tag. Waiting for feedback that merging this would actually make a significative difference in big projects if they have time to try it out.
The main lesson is that you should set-up your build environment to use a complete LaTeX distribution dating back to TeXLive 2019 or more ancient. Avoid newer LaTeX (we can see a 50% increase in build time from 2019 to 2024 regarding sphinx.pdf !!).
Sphinx uses very solid LaTeX packages most of them not having changed for a decade or more and is has very substantial internal LaTeX coding for code-blocks, tables, and admonitions (inclusive of recent #12508 supporting colourful admonitions). It does not especially benefit from being run on current LaTeX. It was better in the past, as everyone knows.
Build times with pdflatex not xelatex for our own Sphinx docs (still on a very old laptop). Timings do not include make latex but only the PDF build time after it (make -C <BUILDDIR> all-pdf).
TL 2019 (pdflatex):
real 0m54.658s
user 0m52.729s
sys 0m1.382s
TL 2024 (pdflatex):
real 1m11.519s
user 1m8.189s
sys 0m2.808s
as mentioned in my previous comment, I have a slight doubt about whether my TL2019 binaries were self-compiled with optimization flags for speed, but I do not think so, and my current understanding is that the decreased performance is due to additional overhead from recent LaTeX macro layer, not from the binaries.
EDIT: to dispel my doubts I used the 2019 pdftex binary together with the 2024 LaTeX distribution and here is the result: (pdflatex 2019 binary used with 2024 LaTeX macro layer and packages)
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
(./sphinx.tex
LaTeX2e <2024-06-01> patch level 2
L3 programming layer <2024-05-27>
(./sphinxmanual.cls
....huge console output...
real 1m12.331s
user 1m10.027s
sys 0m1.918s
This shows that the faster speed with TeXLive 2019 is not due to some special binary build I did from sources with compiler flags for speed. It is due to the extra stuff loaded in the LaTeX macro layer and the extra processing it has added. (it is not from used LaTeX packages, they must have very little changed).
To test if your project using latex_engine='xelatex' would benefit from this PR, it is not even needed to pull this branch.
- execute
make latex - cd to the latex build directory and therein do
make clean - you can now do
time make all-pdfand record the result - then do again
make clean - and
time make all-pdf LATEXMKOPTS="-xelatex"
The behavior at step 5 is what this PR makes default. instructions given for OSes supporting make and passing env vars to it on the command line like I did above.