asv

asv copied to clipboard

Hi,

One issue with running benchmarks is the need for a consistent hardware environment, which reduces the ability for people use things like ASV. I've spent a bunch of time investigating one solution, Cachegrind, which is sucessfully used by SQLite, though not many other projects. Basically, Cachegrind can give you consistent measures of instruction counts, L1 misses, and LL (L3, L2 isn't included) which can be combined into a single, somewhat accurate measure of performance, and consistently across different hardware.

You can see the full write-up here: https://pythonspeed.com/articles/consistent-benchmarking-in-ci/

I am not sure I would trust this as the only measure for scientific computing performance without some long-term measurements comparing it to results on stable hardware, but it's definitely a useful and very robust metric that you can run any random GitHub Actions, GitLab CI, etc., runner and still get consistent results.

Would you be interested in adding this to ASV?

Would you be interested in adding this to ASV?

I, as a asv user, certainly.

But I'm not sure it's possible. How do you plan to convert cachegrind's output to the asv timings?

Measuring things other than time makes sense.

Cachgrind works via emulation right? So this would be problematic for projects that benchmark accelerators like SIMD intrinsics. I wonder if perf or something like it would be both more performant and more predictive.

If you're using perf you get much more accurate numbers, yes... but you have to run on fixed hardware. I would expect that Cachegrind does actually reflect SIMD (you run fewer instructions if you're using SIMD), but it's definitely limited in its realism.