is `@benchmarkset` usable ?

[filchristou]

@benchmarkset is nowhere in the documentation and can only be found in the the API Reference.

Is there a reason why ? Like is it obselete or maybe "too new" ?

Anyway, I think it's also malfunctioning. E.g., I would expect the refdocs example to produce 5 cases but it only produces 1:

julia> @benchmarkset "suite" for k in 1:5
           @case "case $k" rand($k, $k)
       end
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 1-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)

I would expect it to be similar with the following code block:

julia> bg = BenchmarkGroup([], "suite" => BenchmarkGroup([]));

julia> foreach(1:5) do k
           bg["suite"]["case $k"] = @benchmarkable rand($k, $k)
       end

julia> bg
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 5-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 2" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 4" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 3" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 1" => Benchmark(evals=1, seconds=5.0, samples=10000)

System

Julia Version 1.9.4 Commit 8e5136fa297 (2023-11-14 08:46 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 12 × 12th Gen Intel(R) Core(TM) i5-1235U WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-14.0.6 (ORCJIT, alderlake) Threads: 1 on 12 virtual cores Environment: JULIA_IMAGE_THREADS = 1

[32113eaa] PkgBenchmark v0.2.12

similar discussion in https://discourse.julialang.org/t/use-tags-setup-teardown-with-the-benchmarkset-macro/98936/2