PkgBenchmark.jl icon indicating copy to clipboard operation
PkgBenchmark.jl copied to clipboard

Benchmarkables named like tuples are missing from the results table

Open adrhill opened this issue 6 months ago • 0 comments

In our benchmark suite, @gdalle and I had several tests named similarly to named-tuples:

N = 2
K = 3
SUITE["Foo"]["(N=$N, K=$K)"] = @benchmarkable foo($x)

In the table generated by PkgTemplate's export_markdown, benchmarks like this get renamed to the ID ["Foo", (:(N = 2), :(K = 3))]. My best guess is that the string "(N=$N, K=$K)" somehow gets parsed as a named tuple.

The real problem is that all benchmarks with named-tuple-like names get excluded from the comparison table, regardless of improvements or regressions.


We ran into this issue here, where none of our ["bench_jogger.jl", "Hessian", ...] benchmarks appeared in the comparison. However, they did appear in the baseline and target sections with noticeable improvements, as shown in these screenshots. Note that all of them had named-tuple-like names.

adrhill avatar Aug 12 '24 17:08 adrhill