nvbench icon indicating copy to clipboard operation
nvbench copied to clipboard

Allow partial comparison in `nvbench_compare.py`

Open bernhardmgruber opened this issue 2 months ago • 1 comments

When I run the same benchmark, once full and once a subset (to save execution time):

./bin/cub.bench.transform.babelstream.base -d 0 --stopping-criterion entropy --json base.json
./bin/cub.bench.transform.babelstream.base -d 0 -a "T{ct}=I16" -a "Elements{io}[pow2]=28" --stopping-criterion entropy --json new.json

and then compare the results:

PYTHONPATH=./_deps/nvbench-src/scripts python3 ./_deps/nvbench-src/scripts/nvbench_compare.py ./base.json  ./new.json
['./base.json', './base_bif.json']
# Summary

- Total Matches: 0
  - Pass    (diff <= min_noise): 0
  - Unknown (infinite noise):    0
  - Failure (diff > min_noise):  0

I get an empty comparison results. I think the script should report the result for each benchmark which is present in both JSON files and only skip the ones present in only one JSON file.

The current workaround is to create a partial base.json as well, which is 100% unnecessary.

bernhardmgruber avatar Dec 10 '25 09:12 bernhardmgruber

Here are two example files: base.json base_bif.json

bernhardmgruber avatar Dec 10 '25 10:12 bernhardmgruber