Possible improvements in graphing

[aldanor]

From what I gather and from what I've tried, it's currently possible to render the results for each benchmark (i.e. a row) into an svg file using pygal/pygaljs.

There's a few problems with this:

• pygal is far from being a conventional graphing library; it's not shipped with conda and is a bit of pain to build (let alone test); has heavy deps like cairosvg etc

• if you run a parametrized test and want to compare the results (i.e. by putting them in the same benchmark group), the images will still be generated for each separate row, which could be quite meaningless. E.g., if you have a test suite like this:

  @pytest.mark.benchmark(group='read')
  def test_read(benchmark, method):
      benchmark(read(method=method))
  
  @pytest.mark.benchmark(group='write')
  def test_write(benchmark, method):
      benchmark(write(method=method))
  

  where you want to benchmark different read/write methods and method is a parameterized fixture taking on 10 values, this will generate 20 images. However, it would be nice if it could generate just 2 images where you could compare the times and confidence intervals visually (one per group).

• I'm fairly certain this could be nicely done e.g. with matplotlib/seaborn (http://stanford.edu/~mwaskom/software/seaborn) -- if you've never used either I could give a hand with this

[ionelmc]

I wanted to specifically avoid matplotlib because it's a very heavy dependency. It also lumps in numpy and other stuff.

Pygal is very good here as doesn't have any dependencies. The cairosvg thing is completely optional (it's an extra).

Matplotlib plots could make very good results but I think this would be better served with a plugable histogram renderer (so you can do whatever you need).

An example new hook:

def pytest_benchmark_generate_histogram(config, benchmarks, path):
    for name, data in benchmarks:
        ...

[aldanor]

Anyone doing data science in Python would almost surely have both numpy and matplotlib installed; unlike pygal these are also shipped by package distributions like homebrew and Python distributions like conda, making them extremely accessible.

Sort of like pygal/pygaljs are optional dependencies for a specific task, maybe these could be optional dependencies too? But really, this question was more about plotting grouped benchmarks rather than the backend. I could probably sketch a matplotlib version if you want.

[ionelmc]

There are 3 kinds of output from pytest-benchmark:

• simple result table for current run
• compared result table of current run to another run (when adding --benchmark-compare=other-run)
• complete history (--benchmark-histogram)

I like the idea of having different rendering, but pluggable. Eg: matplotlib could be an extra, just like pygal - and you choose what backend you want.

Ignoring #26 for a moment, maybe a --benchmark-histogram-backend=[pygal|matplotlib] argument?

[aldanor]

Yep that sound good, as long as there's also an option to --benchmark-group-histograms or something like this?

[ionelmc]

What would --benchmark-group-histograms do?

[aldanor]

Having one plot per benchmark group and not per each benchmark making possible to actually compare things visually within one run. (sorry if I missed something in this discussion)

[ionelmc]

I see, yes, we should have that.

The current histogram generating code needs some refactoring for that though ... you said you'd contribute a matplotlib backend at least right? :grin:

[aldanor]

Yea sure I could help with matplotlib backend when there's a way to plug it in and another implementation as a reference (e.g. pygal). Maybe the refactoring should be done first then?