Stop testing a file on first fail

[james10424]

What's the problem this feature will solve?

This allows me to gather all failed files quicker, I don't need to know whether the remaining tests passed in this file.

Describe the solution you'd like

This will be equivalent to the -x option, but on a per-file basis. That is, pytest stops testing this file if it has one failed test.

This can save time when running on clouds like Github Action, if it fails fast, then I'll be charged less, knowing I have to re-run them.

Alternative Solutions

I'm not sure which plugin can solve this problem.

Additional context

[tirkarthi]

As a workaround you can probably have a fixture that stores the module name on first failure to a list and raise skip for rest of the files of the same module name. It also will not work on xdist since the list is at module level and you might need to store it somewhere reliable for parallel processing.

This can save time when running on clouds like Github Action, if it fails fast, then I'll be charged less, knowing I have to re-run them.

-x is useful in this case. There is also --max-fail=n to fail on n number of failures. pytest also has other modes like --last-failed where if you can persist .pytest-cache folder between rerun only failed tests per PR.

  -x, --exitfirst       exit instantly on first error or failed test.
  --runxfail            report the results of xfail tests as if they were not
  --lf, --last-failed   rerun only the tests that failed at the last run (or all
                        if none failed)
  --ff, --failed-first  run all tests, but run the last failures first.
  --lfnf={all,none}, --last-failed-no-failures={all,none}
                        which tests to run with no previously (known) failures.
  --sw, --stepwise      exit on test failure and continue from last failing test
                        ignore the first failing test but stop on the next
                        failing test
                        (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
                        failed tests. Default is 'all'.
  --maxfail=num         exit after first num failures or errors.
  --doctest-report={none,cdiff,ndiff,udiff,only_first_failure}
                        failure
  --doctest-continue-on-failure
                        failure
                        xfail_strict=True -o cache_dir=cache`.
  -f, --looponfail      run tests in subprocess, wait for modified files and re-
                        run failing test set until all pass.
  xfail_strict (bool):  default for the strict parameter of xfail markers when

import pytest


skip_modules = []


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()
    setattr(item, "rep_" + rep.when, rep)


@pytest.fixture(autouse=True)
def skip_file_on_fail(request):
    mod_name = request.module.__name__

    if mod_name in skip_modules:
        pytest.skip(f"Skipping rest of tests in {mod_name}")

    outcome = yield

    if request.node.rep_call.failed:
        skip_modules.append(request.module.__name__)

[The-Compiler]

Not sure if this should be closed, @Zac-HD - it seems like an useful proposal at first sight.

[pansila]

Worth a mention in the doc.