RFC: integration with pytest(-benchmark)

[nzjrs]

I've got a relatively large project with many benchrmarks written as pytest fixtures and benchmarks collected with pytest-benchmark

Would it be feasible to use pytest to discover and execute these benchmarks? In particular, I find the operation of @pytest.mark.parameterize to work very nicely with pytest-benchmark and allow easy comparative bench-marking. Do you think it is feasible to integrate with pytest in this way?

[pv]

Feasible in principle, yes.

However, at the moment I suspect it's not a good idea to burden the development with supporting multiple benchmark runners. If you need integration now, a simpler solution is to whip up a script that converts pytest-benchmark json files to asv json --- that's probably doable in a few hours.

[nzjrs]

cool, is the format of the asv json files defined somewhere?

[klauer]

While it's very far from perfect, I worked on some code that allowed for generation of asv-compatible json using pytest-benchmark. Here's some links:

Example generated from pytest-benchmark+asv, ugly asv/pytest-benchmark bridge code, and required lines in conftest

If there's some interest, maybe this could be cleaned up and put in a package of its own.

[1heart]

For those searching, it looks like RapidsAI has some sort of pytest-benchmark + asv solution using a CLI-based asv "database" viewer.

[vyasr]

@pv I work on RAPIDS and my colleague @rlratzel is the maintainer of the asvdb package mentioned in the previous comment. Our use case is basically what's been described here: we have a large test suite already written with different tools (including pytest-benchmark) and we wanted to take advantage of asv's awesome viewer features without having to rewrite our entire test suite. Our solution is basically what you proposed a few comments ago on this thread. We developed asvdb as a programmatic means for interacting with asv's JSON database, and we separately built the rapids-pytest-benchmark pytest plugin that builds on top of pytest-benchmark and asvdb to allow running a pytest-benchmark and immediately dumping to asv by adding an extra command line argument.

I agree that full pytest-benchmark integration with asv is probably out of scope for asv at this stage. However, I assume that asv already has some sort of internal machinery for working with its JSON format that probably overlaps with our asvdb. Would you be open to maybe exposing some of those internals or allowing us to upstream asvdb into asv so that there's a canonical programmatic method for interacting with asv's JSON "database"? Having that feature baked directly into asv would be preferable to having it reside in a third-party, not officially supported package, and it's likely that you have internals that asvdb is largely replicating in order to write these files. If you're interested in, we could collaborate on getting asvdb (or a subset of it) merged into asv. We'd be happy to maintain the rapids-pytest-benchmark plugin separately since the actual pytest-benchmark->asv conversion seems out of scope for asv and would be straightforward to maintain if the asv-writing functionality was part of asv.