benchmark icon indicating copy to clipboard operation
benchmark copied to clipboard

[FR] Add the ability to dry-run benchmarks

Open ldionne opened this issue 6 months ago • 10 comments

Is your feature request related to a problem? Please describe. As part of the libc++ test suite, we use GoogleBenchmark to write micro-benchmarks for various parts of the library. Running those benchmarks takes a lot of time, so we don't run them by default. However, this has the downside that benchmarks can easily rot and become invalid.

Describe the solution you'd like We would like the ability to "dry-run" benchmarks. Basically, we'd like an argument that can be passed to a benchmark that forces it to do only one iteration and then return. This would allow us to build and dry-run the benchmarks in our normal testing configuration, and to run the full benchmarks when we actually want to.

There are a few different ways this could be achieved depending on what the maintainers think is best:

  • Add a new --benchmark_dry_run=True argument (or similar, but basically a boolean that represents a dry-run)
  • Add the ability to control the maximum number of iterations of a benchmark like --benchmark_max_iterations=N
  • Add the ability to bound the time taken for each benchmark, like --benchmark_max_time=TIME

Any of these APIs would solve our problem.

ldionne avatar Jul 31 '24 16:07 ldionne