benchmark

benchmark copied to clipboard

Null hypothesis test discrepancy

9

https://github.com/google/benchmark/blob/49aa374da96199d64fd3de9673b6f405bbc3de3e/tools/gbench/report.py#L212 As far as I understand, in google benchmark `compare.py` script we set: - The null hypothesis as the baseline and the contender to have the same results. - The...

vicentebolea

[FR] Allow benchmarks to report their parameterization structurally to JSON

10

**Is your feature request related to a problem? Please describe.** The `benchmark::RegisterBenchmark()` API supports adding benchmarks that are parameterized with arbitrary values, including custom names, lambdas, arbitrary objects, etc. The...

matta

Support --benchmarks_filter in the compare.py 'benchmarks' command

3

Previously compare.py ignored the --benchmarks_filter argument when loading JSON. This defeated any workflow when a single run of the benchmark was run, followed by multiple "subset reports" run against it...

matta

[windows] Fix for background colours besides black

1

Fix for issue https://github.com/google/benchmark/issues/1487.

gvanem

[windows] Terminal background colour

8

Hello folks, my first issue. The function `ColorPrintf()` ass-u-mes that all users of this library have terminals with a black background colour. I.e. using it in [Microsoft's STL](https://github.com/microsoft/STL.git) library and...

gvanem

[FR] Support compare.py benchmarks a.json b.json --benchmark_filter=blah

6

**Is your feature request related to a problem? Please describe.** I've got an a.json and a b.json that took a while to collect due to lots of benchmarks and lots...

matta

Rewrite complexity_test to use (hardcoded) manual time

4

This test is fundamentally flaky, because it tried to read tea leafs, and is inherently misbehaving in CI environments, since there are unmitigated sources of noise. Fixes https://github.com/google/benchmark/issues/272 Fixes https://github.com/google/benchmark/issues/1758

LebedevRI

If you do `UseManualTime()`, and `Complexity()`, the Big-O will be based on the CPU time, not the real time which is your manual time...

LebedevRI

[BUG] With NDEBUG, early-returning without SkipWithError leads to indefinite retries

5

**Describe the bug** With NDEBUG, early-returning without SkipWithError leads to indefinite retries **System** Which OS, compiler, and compiler version are you using: - OS: ```6.5.13-1rodete2-amd64 (building for Android cuttlefish)``` -...

adrianroos

Add min rel accuracy stopping criterion

9

This is work in progress to try to add a statistical stopping criterion. While the existing statistics features seek to characterise the variability between repetitions of the benchmark, the statistical...

maartenarnst