PkgBenchmark.jl
PkgBenchmark.jl copied to clipboard
Benchmarkables named like tuples are missing from the results table
In our benchmark suite, @gdalle and I had several tests named similarly to named-tuples:
N = 2
K = 3
SUITE["Foo"]["(N=$N, K=$K)"] = @benchmarkable foo($x)
In the table generated by PkgTemplate's export_markdown
, benchmarks like this get renamed to the ID ["Foo", (:(N = 2), :(K = 3))]
.
My best guess is that the string "(N=$N, K=$K)"
somehow gets parsed as a named tuple.
The real problem is that all benchmarks with named-tuple-like names get excluded from the comparison table, regardless of improvements or regressions.
We ran into this issue here, where none of our ["bench_jogger.jl", "Hessian", ...]
benchmarks appeared in the comparison. However, they did appear in the baseline and target sections with noticeable improvements, as shown in these screenshots. Note that all of them had named-tuple-like names.