ENH: Merging results from sharded benchmarks

[jackie-pc]

Great work team - asv looks nice.

We are looking into using ASV for tracking benchmarks for the reflex project.

Some of the planned benchmarks can take a long time to run (say minutes), and we would ideally want to shard them out. E.g. one job runs suite A, another job runs suite B, another job run suite C, etc.

Would this be a sensible or supported workflow for asv? In particular I'm looking at the results/ dir. How would one merge results/ dirs produced by different jobs? With a valid merge operation, I can fan-in all the results in a single "publish" job at the end. This gives me a single static site to browse.

I am looking at results/benchmark.json in particular.... Each run seems to produce one of these. Is it as simple as just concatenating the list within all the benchmark.json's? :)