pybamm-team
Add `figure tests` for plotting functions
Currently, PyBaMM lacks automated figure tests to verify the correctness of plots generated by QuickPlot or other plotting functions. This can lead to unintended changes in the plots and make it difficult to verify whether modifications to these functions are correctly implemented. Adding figure-based tests using
pytest-mpl or a similar approach will help detect unintended changes in plot outputs.
Sounds good to me – thanks for opening this issue publicly following our discussion, @medha-14! I cannot comment on how how long it can take or how a framework for this can be set up, as you are more familiar with it. I can only say that this is very cool, and I would be open to a PoC PR, after which we can decide whether we want to keep it.
The decision would likely be based on the implementation, i.e. how the code is written, and on the ease of maintenance:
- how easy it will be to debug failing image comparisons
- how do failures show up in the tests
- are there hints on why they fail
- is there a chance they will fail often, i.e., will they be flaky, etc.
- what parts of the plotting functionality are tested
If you have answers to these questions or are working on assembling them, please feel free to comment them below whenever you do (or in the PR thread as you wish).
Linking the SunPy implementation and relevant references here:
- https://github.com/sunpy/sunpy/blob/386c970302ad5cedd69e5a88b518b48a5daa4f3f/sunpy/tests/helpers.py#L86-L113
- https://github.com/sunpy/sunpy/blob/386c970302ad5cedd69e5a88b518b48a5daa4f3f/sunpy/tests/figure_hashes_mpl_382_ft_261_astropy_600_animators_111.json
- https://github.com/matplotlib/pytest-mpl
I am currently working on this and will be setting up a PoC PR in a few days time, which will be focused on evaluating how well this feature handles image comparisons, debugging failures, and also how easy it will be to maintain. I'll share my findings and would love to get feedback along the way.