Have an option to implement new test result / outcome

[gs202]

I've encountered a situation where i need to defer 2 types of skip statuses (in addition to xfail) and perform different actions + reporting at the end of the test.

Suggestion: Have an option to create new test status outcome (e.g. pytest.feature_not_ready), that inherent from pytest OutcomeException class and allows us to increase the diversity of test results.

For example:

@pytest.fixture
def check_precondition(precondition):
    if not precondition:
        do_action_a()
        pytest.skip(f"precondition {precondition} not met, can't continue the test")

@pytest.fixture
def check_exisiting_bugs(bugs_list, metafunc):
    test_name = metafunc.definition.name
    for bug in bugs_list:
        if bug.test_name == test_name:
            do_action_b()
            pytest.feature_not_ready(f'will not run test {test_name} with opened bug {bug}')

I've tried to use the pytest_exception_interact and create a custom exception to be raised for the new test outcome, using this exception control/add my actions. The problem with this solution is that the exception is caught after test result is determined → test is failed and i can't control the result (e.g. skip #2). If at the end of the exception class, the pytest.skip will be raised - the exception itself won't be caught by pytest_exception_interact and it will be identical to regular skip test result (which i try to avoid and add another)

[webknjaz]

ref https://github.com/pytest-dev/pytest/issues/7327

[jbrockmendel]

Similar use cases, it'd be nice to distinguish between

-"xfail that represents a bug" vs "xfail that we're waiting for an upstream dependency to fix"

• "pytest.skip for traditional reason e.g. 'not supported on windows'" vs "pytest.skip bc it is too messy to exclude this combination of fixture params"

[RonnyPfannschmidt]

Im working on part of this in 2 plugins

Pytest-external-blockers and pytest-manual,

Jira/githib/gitlab based test gating will hopefully happen this year

[ghost]

I would like to see this as well. I'd prefer to not fork or patch pytest itself. Extra outcomes would be very welcome since we report up to testrail which supports a good number of result types.