tabularray Write some benchmark tests

We need some benchmark tests for future performance improvement.

Feb 22 '24 01:02 lvjr

The problem is that l3benchmark is not so accurate, even on Linux. Last year I ran the same benchmark test several times on Overleaf TeXLive 2022 and got the following output:

  0.0139 seconds (4.7e4 ops)
  0.425 seconds (1.58e6 ops)
  0.00421 seconds (1.15e4 ops)
  0.446 seconds (1.01e6 ops)
  0.00366 seconds (7.61e3 ops)
  0.433 seconds (8.88e5 ops)
  0.0027 seconds (9.92e3 ops)
  0.368 seconds (1.37e6 ops)
  0.00261 seconds (9.65e3 ops)
  0.423 seconds (1.57e6 ops)
  0.0027 seconds (1e4 ops)
  0.368 seconds (1.36e6 ops)

See also https://github.com/latex3/latex3/issues/1182 .

Feb 22 '24 02:02 lvjr

In above commit, I added a basic benchmark test file. The goal is to add as many interfaces as possible to the file, since this file is also used in regression tests in old TeX Live releases (#525).

Dec 07 '24 12:12 lvjr

Apart from the overall time consumption, may also need to do performance analysis (profiling) to find and probably improve the bottleneck.

Dec 07 '24 18:12 muzimuzhi

Yes, this is just a minor step.

Dec 08 '24 04:12 lvjr

It seems we can also compile the benchmark file with programs in latex-dev.

Dec 10 '24 00:12 lvjr

I did some tests by compiling tblrbenchmark.tex four times with different engines in Windows, and from the results of used time it seems current tabularray is much faster than previous tabularray (v2024A). Why? I can not find the reason from the commit log. :confused:

pdflatex with tabularray 2024A
1       4.785995483398438
2       4.79400634765625
3       4.8070068359375
4       5.654006958007812
pdflatex with current tabularray
1       3.49700927734375
2       3.506988525390625
3       3.49200439453125
4       3.498992919921875
-------------------------
xelatex with tabularray 2024A
1       4.899002075195312
2       4.9639892578125
3       4.896987915039062
4       4.915008544921875
xelatex with current tabularray
1       3.61700439453125
2       3.649002075195312
3       3.671005249023438
4       3.623992919921875
-------------------------
lualatex with tabularray 2024A
1       5.660964965820312
2       5.758621215820312
3       5.668914794921875
4       5.6624755859375
lualatex with current tabularray
1       4.215377807617188
2       4.238525390625
3       4.221893310546875
4       4.2078857421875

Dec 11 '24 01:12 lvjr

... it seems current tabularray is much faster than previous tabularray (v2024A). Why? I can not find the reason from the commit log. 😕

git bisect the commits between them?

Dec 11 '24 02:12 muzimuzhi

From now on every benchmark result will be published to the following urls in gh-pages:

for texlive 2023: https://lvjr.github.io/tabularray/benchmark/2023
for texlive 2024: https://lvjr.github.io/tabularray/benchmark/2024

The benchmark.lua script compares running time between latest tabularray and previous tabularray (v2024A at present).

Dec 11 '24 07:12 lvjr

Benchmark page for texlive 2022 was also added: https://lvjr.github.io/tabularray/benchmark/2022 .

Dec 11 '24 07:12 lvjr

... it seems current tabularray is much faster than previous tabularray (v2024A). Why? I can not find the reason from the commit log. 😕

git bisect the commits between them?

Reset to commit https://github.com/lvjr/tabularray/commit/9d911e2f6bf73bb58ab75981641719fdce782519 (set measure=vbox as default with varwidth library (#540))

pdflatex used time 1    old = 4.7830047607422   new = 3.4969940185547
pdflatex used time 2    old = 4.77099609375     new = 3.4980010986328
pdflatex average time   old = 4.7770004272461   new = 3.4974975585938   ratio = 0.73215349503538
xelatex used time 1     old = 4.9150085449219   new = 3.6349945068359
xelatex used time 2     old = 4.9759979248047   new = 3.6380004882812
xelatex average time    old = 4.9455032348633   new = 3.6364974975586   ratio = 0.73531394579284
lualatex used time 1    old = 5.6410217285156   new = 4.1806182861328
lualatex used time 2    old = 5.6429595947266   new = 4.2891845703125
lualatex average time   old = 5.6419906616211   new = 4.2349014282227   ratio = 0.75060411868988

Reset to commit https://github.com/lvjr/tabularray/commit/486cccda6e98e9f15dd711d738e86e14dfc978b4 (Merge branch 'main' into issue179)

pdflatex used time 1    old = 4.7940063476562   new = 4.8269958496094
pdflatex used time 2    old = 4.7949981689453   new = 5.3399963378906
pdflatex average time   old = 4.7945022583008   new = 5.08349609375     ratio = 1.060276087043
xelatex used time 1     old = 4.9029998779297   new = 4.9349975585938
xelatex used time 2     old = 4.8890075683594   new = 4.9359893798828
xelatex average time    old = 4.8960037231445   new = 4.9354934692383   ratio = 1.0080657099804
lualatex used time 1    old = 5.6476287841797   new = 5.6497802734375
lualatex used time 2    old = 5.6314086914062   new = 5.7043762207031
lualatex average time   old = 5.639518737793    new = 5.6770782470703   ratio = 1.0066600557643

The difference is that we load varwidth library in tblrbenchmark.tex and now measure=vbox is the default. Therefore the above result shows that measure=vbox is much faster than measure=hbox.

Dec 11 '24 10:12 lvjr

This deserves to be mentioned in "Tips and Tricks" chapter.

Dec 11 '24 10:12 lvjr