benchmark-compare no longer defaulting to comparing against the previous run

[jab]

My test suite passes --benchmark-compare with no value specified, so that it defaults to the latest saved run. Upgrading from 3.0.0 to 3.1.0a1 broke this such that pytest-benchmark now always compares the current run against itself, rather than against the previous run.

[jab]

What would really be helpful is if there were a way to set up pytest-benchmark and Travis-CI so that:

• the benchmark data for the latest build of master were always saved with an alias like latest-master, and would compare against the previous master benchmark data
• when running tests on Travis for a build of a topic branch, always compare against the latest-master benchmark data

This way, for projects where master is considered the stable branch, its benchmark data is always used as the reference when comparing benchmarks run during the build of a topic branch.

[ionelmc]

Or you could only save bnchmark data from master builds - it sounds like you don't need to save bench data from topic branche builds?

The comparing against itself sounds like a bug (looking into it),

[ionelmc]

I'm considering adding support for saving with something like --benchmark-save=latest-{branch}. That should support your usecase.

Currently --benchmark-save=NAME saves into STORAGE-PATH/counter_NAME.json which is quite different.

We could have --benchmark-save=NAME_OR_FMT with this synopsis:

Save the current run into 'STORAGE-PATH/counter_NAME.json' or, if it contains curly brackets, 'FMT.json' formatted against a dict containing storage_path, counter, branch, id (revision number), short_id, project, dirty, node, processor, machine, python_compiler, python_implementation, python_implementation_version, python_version, python_build, release, system and cpu.

Hmmm, @antocuni critique this please.

Also, @jab the compare bug should be fixed.