asv

asv copied to clipboard

I've been having a look to the configuration files of some of the main projects using asv (pandas, xarray, scikit-learn, pymc3, scikit-image...), and I don't see any of them using environments, a matrix of dependencies or different Python versions. At least in the config file in the repo.

Feels to me like asv has a decent amount of complexity which is not widely used, and that could anyway be handled easily outside of asv (creating and activating the environments manually).

Even things like running asv for a sequence of commits could be done easily with bash (e.g. loop over git log --pretty=format:"%h" 1b5338e9...976b348e checkout the commit with git, and run for the working directory), making asv agnostic of source control systems, simplifying the code to a very lightweight library, and making maintenance very easy, with the same functionality (just less shortcuts IMHO).

Am I missing something? How does other people feel about moving in this direction?

I've been having a look to the configuration files of some of the main projects using asv (pandas, xarray, scikit-learn, pymc3, scikit-image...), and I don't see any of them using environments, a matrix of dependencies or different Python versions. At least in the config file in the repo.

This is our approach for Iris too - to defer to a custom environment creation script. I had always assumed that there was another part of the Python community that highly valued the matrix approach, even if many of us don't.

---

Even things like running asv for a sequence of commits could be done easily with bash

ASV provides some very valuable intelligence in this regard. I can think of the following features that depend on specialist commit handling, all of which we have found valuable in the last year. There are no doubt more features that we haven't come across yet.

• The --interleave-rounds argument
• The asv continuous command
• asv publish determining a linear commit history

I appreciate there's a compromise where you don't want to support conveniences for everyone's specific workflows, but removing the 'commit intelligence' and expecting users to work it out themselves feels like a step too far in the light-weight direction.

The 'main projects' you mention have a lot of energy behind them, with a lot of very smart people working on them. It might be easy to forget how benchmarking is very difficult to get right!

ASV's shortcuts can be valuable in making benchmarking a bit easier for projects that have less experience. But yeah I appreciate there might not be enough resource to support all the shortcuts.

Thanks for the feedback @trexfeathers

Just to add more information on where this comes from. asv is quite a huge project, with an old codebase, and with almost no maintenance time spent on it. We've got a pandas grant that we're using to help with it, and that won't last long. The two options I see if things don't change are:

• We all stay in asv 0.4 or 0.5 forever (or slowly migrate to other tools)
• We leave asv in a position where it can be maintained with the available resources

Of course there has been a lot of energy on developing asv, and it's nice to have all features we need in a nice way. But since the cost of having so many features seems to be a problem, I want to open this discussion on whether it makes sense to get rid of things that (at least to me) are not key, but that add a huge amount of the complexity of the project. It's great to know about how different users use asv and what is important for them, since each of us individually has a partial and biased point of view. And I really appreciate your feedback. Let's see if there are other points of view, and what the best solution can be.

We've got a pandas grant that we're using to help with it, and that won't last long.

Ah I hadn't realised it was short-term!

Well, if there are features that need to be culled, a docs page with examples of how to 'do it yourself' might be a good way to compensate 🙂

Can I ask: if environment and commit management are removed, what are the remaining reasons to use ASV over pytest-benchmark?

I'm sure there are good reasons, but I need educating 🙂

After the refactor of asv into asv and asv_runner in https://github.com/airspeed-velocity/asv/pull/1287 the components left in asv are:

• Environment creation
• Collection of commits / tags etc.
• Running benchmarks (now in asv_runner)
• Aggregation of results
• Dashboard generation

Given that, asv_runner is a bit closer in spirit to pytest-benchmark in that it is meant for single benchmarks. IMO there isn't a good reasonto deprecate the isolation and environment creation logic asv handles.