--rerun-except retries the test once

[srgsanky]

I used --rerun-except and was expecting the test to never be retried on the given regex. But I found that the test was retried once. Here is a sample test case test_demo.py

import pytest


class SampleException(Exception):
    pass


attempt = 0


@pytest.mark.flaky(reruns=5)
def test_example():
    global attempt
    attempt = attempt + 1
    print(f"Running test_example {attempt}")
    raise SampleException()

requirements.txt

pytest==7.2.1
pytest-rerunfailures==11.1.1

Running

python3 -m pytest  --cache-clear -s *.py -k test_example --rerun-except 'SampleException'

produces

========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.9.6, pytest-7.2.1, pluggy-0.13.1
rootdir: /Volumes/workplace/PytestSample
plugins: rerunfailures-11.1.1, retry-1.2.1
collected 1 item                                                                                                                                                                                         

test_demo.py Running test_example 1
RRunning test_example 2
F

================================================================================================ FAILURES ================================================================================================
______________________________________________________________________________________________ test_example ______________________________________________________________________________________________

    @pytest.mark.flaky(reruns=5)
    def test_example():
        global attempt
        attempt = attempt + 1
        print(f"Running test_example {attempt}")
>       raise SampleException()
E       test_demo.SampleException

test_demo.py:16: SampleException

==================================================================================== the following tests were retried ====================================================================================
        test_example failed on attempt 1! Retrying!
        Traceback (most recent call last):
          File "/Volumes/workplace/PytestSample/test_demo.py", line 16, in test_example
            raise SampleException()
        test_demo.SampleException

        test_example failed after 2 attempts!
        Traceback (most recent call last):
          File "/Volumes/workplace/PytestSample/test_demo.py", line 16, in test_example
            raise SampleException()
        test_demo.SampleException

======================================================================================== end of test retry report ========================================================================================

======================================================================================== short test summary info =========================================================================================
FAILED test_demo.py::test_example - test_demo.SampleException
====================================================================================== 1 failed, 1 retried in 0.02s ======================================================================================

Is this the expected behavior? Is it possible to avoid even the single retry?

[icemac]

Thank you for your report. Does your example behave the same when using pytest-rerunfailures==11.0? (There were some changes in the last feature release where we currently face some regressions.)

[srgsanky]

Yes, the behavior is the same with pytest-rerunfailures==11.0.

======================================================================================================== test session starts ========================================================================================================
platform darwin -- Python 3.9.6, pytest-7.2.1, pluggy-0.13.1
rootdir: /Volumes/workplace/PytestSample
plugins: rerunfailures-11.0, retry-1.2.1
collected 1 item                                                                                                                                                                                                                    

test_demo.py Running test_example 1
RRunning test_example 2
F

============================================================================================================= FAILURES ==============================================================================================================
___________________________________________________________________________________________________________ test_example ____________________________________________________________________________________________________________

    @pytest.mark.flaky(reruns=5)
    def test_example():
        global attempt
        attempt = attempt + 1
        print(f"Running test_example {attempt}")
>       raise SampleException()
E       test_demo.SampleException

test_demo.py:16: SampleException

================================================================================================= the following tests were retried ==================================================================================================
        test_example failed on attempt 1! Retrying!
        Traceback (most recent call last):
          File "/Volumes/workplace/PytestSample/test_demo.py", line 16, in test_example
            raise SampleException()
        test_demo.SampleException

        test_example failed after 2 attempts!
        Traceback (most recent call last):
          File "/Volumes/workplace/PytestSample/test_demo.py", line 16, in test_example
            raise SampleException()
        test_demo.SampleException

===================================================================================================== end of test retry report ======================================================================================================

====================================================================================================== short test summary info ======================================================================================================
FAILED test_demo.py::test_example - test_demo.SampleException
=================================================================================================== 1 failed, 1 retried in 0.02s ====================================================================================================