asv

asv copied to clipboard

If I create a new benchmark I generally want to run the benchmarks for all existing releases (in my case git tags). The -b optional already helps to select only the new test but it's either not easy to run on all tags or I missed the option.

Nevertheless: I think it would be really nice to have an "ALL_TAGS" option equivalent to the already existing "ALL", "EXISTING" and "NEW" option for range in asv run.

That would be a really great feature!

I think this can already be done with asv run $(git tags -l | xargs) for git and asv run 'tag()' for mercurial.

That doesn't seem to work for me. If I try it errors

asv/> asv run $(git tags -l | xargs)
git: 'tags' is not a git command. See 'git --help'.

Did you mean one of these?
    stage
    tag

If I instead run asv run $(git tag -l | xargs) (tag instead tags) it simply runs the latest commit, and if I do asv run $(git tags -l | xargs) v1.0.0..master it runs all commits, not just tags.

The problem with this is that it looks like asv runs every commit between tag x until tag x+1. So running asv run v1.0.0 runs all commits from v1.0.0 until the last commit before the next tag, say v1.0.1.

Ok maybe asv run $(git tag -l | awk '{ print $1 "^.." $1 }' | xargs) ?

Unfortunately not, it just runs the latest commit for me if I try the command.

asv run "git tag -l | awk '{ print $1 "^!" }'"is the incantation, but as seen above, not so easy to recall...

asv/> asv run "`git tag -l | awk '{ print $1 "^!" }'`"
asv run "`git tag -l | awk '{ print $1 "^"`git tag -l | awk '{ print $1 "^" }'`" }'`"
bash: command substitution: line 1: unexpected EOF while looking for matching `''
bash: command substitution: line 2: syntax error: unexpected end of file
bash: command substitution: line 1: unexpected EOF while looking for matching `"'
bash: command substitution: line 2: syntax error: unexpected end of file
· Fetching recent changes
· Error running /usr/bin/git rev-list --first-parent git tag -l | awk '{ print ^ }'
            STDOUT -------->

And if I run

asv run `git tag -l | awk '{ print $1 "^!" }'`

instead it does not fail, but it only runs the last commit, not the tags.

It works fine in bash 4.4.19.

Ha, interesting. I am on GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu) (Ubuntu 16.04).

Probably it points towards that the addition of extra ALL_TAGS keywords could be useful :)

Additionally, "asv run" probably should accept multiple input arguments, so that one wouldn't need to work hard on getting the quoting right which apparently depends even on bash version...

I do think a ALL_TAGS keyword would be great. But then I have no idea how easy/difficult it is to implement.

Additionally, "asv run" probably should accept multiple input arguments

Totally agree. Because next I though I just get a list of tags with git tag -l | awk '{ print $1 "^!" }' and provide it to asv run. But that doesn't work, so one would have to loop over it.

So thumb up from my side for both, ALL_TAGS and multiple input arguments.

This seems to work for me:

git tag -l | awk '{ print $1 "^!" }' | xargs -L 1 asv run 

is basically just using xargs as a for loop to loop over every tag in turn.