flaky icon indicating copy to clipboard operation
flaky copied to clipboard

Flaky decorators on individual tests have no effect if `--force-flaky` argument is passed

Open tdluk opened this issue 10 months ago • 1 comments

From README.rst:

Pass --max-runs=MAX_RUNS and/or --min-passes=MIN_PASSES to control the behavior of flaky if --force-flaky is specified. Flaky decorators on individual tests will override these defaults.

However, flaky decorators on individual tests do not override these defaults.

To repro the issue, consider the following test file test.py:

import pytest

@pytest.mark.flaky(max_runs=10)
def test_foo():
    assert False

Then run pytest test.py --force-flaky and observe the test was run the default 2 times instead of the 10 specified by the decorator.

tdluk avatar Apr 16 '24 10:04 tdluk

Problem appears to be here:

https://github.com/box/flaky/blob/d29ca76d5a72845428a0716c74f8fa0d40d98532/flaky/flaky_pytest_plugin.py#L252

This means an individual test's decorators won't be translated into flaky attributes, as the test's flaky attributes have already been set to the defaults (or values set in CLI args) here:

https://github.com/box/flaky/blob/d29ca76d5a72845428a0716c74f8fa0d40d98532/flaky/flaky_pytest_plugin.py#L77-L82

tdluk avatar Apr 16 '24 10:04 tdluk