pytest
pytest copied to clipboard
pytest-dev
Option to rerun all tests in file with last-failed
What's the problem this feature will solve?
I do use --last-failed to rerun flaky tests. This works perfectly fine in formal testing of libraries, however when I use the same machinery for jupyter notebook-based tutorials I need to rerun all previous cells, too as tests are not standalone.
(I use the nbval plugin to collect the cells as tests)
My other option would be to use pytest-rerunfailes, but that plugin reruns the failures straight away, and most of my cases are flaky as they query a remote server. So the approach of --last-failed would be preferred, aka to have a second test tun that only runs the problematic ones.
Describe the solution you'd like
Have an option to be used along with --last-failed that would collect all the tests from the files contained a failed test
Alternative Solutions
Additional context
The specific use case creates the need for test dependencies as the jupyter plugin isn't using subtests
At first glance it doesn't correctly genralize
Test items per design are independent and pytest core shouldn't have to handle plugins that don't respect that
I understand the need for this feature, but as @RonnyPfannschmidt mentioned, tests should ideally be independent.
Implementing this workaround in pytest core would likely add more code bloat to an area that's already quite complex—especially with the lastfailed and piecewise plugins providing comparable functionality.
A dedicated plugin for rerunning notebook-related test failures might ultimately deliver a better user experience.
I will close this for now, but thanks for the report regardless @bsipocz!
