nvbench
nvbench copied to clipboard
Allow partial comparison in `nvbench_compare.py`
When I run the same benchmark, once full and once a subset (to save execution time):
./bin/cub.bench.transform.babelstream.base -d 0 --stopping-criterion entropy --json base.json
./bin/cub.bench.transform.babelstream.base -d 0 -a "T{ct}=I16" -a "Elements{io}[pow2]=28" --stopping-criterion entropy --json new.json
and then compare the results:
PYTHONPATH=./_deps/nvbench-src/scripts python3 ./_deps/nvbench-src/scripts/nvbench_compare.py ./base.json ./new.json
['./base.json', './base_bif.json']
# Summary
- Total Matches: 0
- Pass (diff <= min_noise): 0
- Unknown (infinite noise): 0
- Failure (diff > min_noise): 0
I get an empty comparison results. I think the script should report the result for each benchmark which is present in both JSON files and only skip the ones present in only one JSON file.
The current workaround is to create a partial base.json as well, which is 100% unnecessary.
Here are two example files: base.json base_bif.json