nvbench icon indicating copy to clipboard operation
nvbench copied to clipboard

Add benchmark groups/sets/tags

Open pauleonix opened this issue 2 years ago • 3 comments

It would enable a nicer (shorter) way of specifying benchmarks in the CLI interface if one could define groups/sets of benchmarks in the source code.

One way of doings this could be to leverage existing namespaces, i.e. being able to call bench --namespace foo to execute all benchmarks in that namespace (and sub-namespaces). Although I don't know how easy this is to implement compared to just adding a new class.

pauleonix avatar Apr 07 '22 12:04 pauleonix

Using namespaces would also be somewhat limiting as one can't have two namespaces with partial overlap.

pauleonix avatar Apr 07 '22 13:04 pauleonix

What comes to mind is how Catch2 uses tags.

We could adapt this idea to nvbench to look something like:

NVBENCH_BENCH(benchmark).add_tags("[foo][bar]")

You could then specify on the CLI one or more tag strings and it would run only the benchmarks with that tag.

jrhemstad avatar Apr 07 '22 13:04 jrhemstad

I like the idea, and the suggestion to use familiar Catch2 syntax is definitely a good direction to take. I'd been planning to add regex matching to the -b <benchmark string>, but this would provide greater flexibility 👍

alliepiper avatar Apr 08 '22 16:04 alliepiper