Reordering tests with `pytest_collection_modifyitems` hook fails

[michalkret]

Hiya pytest folks! I've got the problem with pytest hooks. The checklist was checked as below:

• [✅] a detailed description of the bug or problem you are having
• [✅] output of pip list from the virtual environment you are using
• [✅] pytest and operating system versions
• [✅] minimal example if possible

pytest version: 8.00 python version: 3.12.2 OS: python:dev-3.12-bookworm Debian container

Issue: I'm trying to reorder execution reorder of my tests using pytest_collection_modifyitems hook as described in: https://docs.pytest.org/en/latest/reference/reference.html#pytest.hookspec.pytest_collection_modifyitems

I added the following code in the top dir conftest.py:

def pytest_collection_modifyitems(config, items):
    grouped_items = {}
    for item in items:
        #  marks based reordering logic, not important for the issue
        grouped_items[key].append(item)

    new_items = []
    for key, items in sorted(grouped_items.items()):
        new_items.extend(items)

    items[:] = new_items

I noticed that tests are not running in the expected order, so I added logging code at the end in the modified hook to double check if it's correct:

    for item in items:
        marks = item.iter_markers()
        test_name = item.name
        marks_str = ", ".join(mark.name for mark in marks)
        logging.info(f"Test '{test_name}' has marks: {marks_str}")

and indeed it is. If I however log items from session.items in pytest_collection_finish(session: pytest.Session) hook my order will not be there anymore. Interestingly enough, moving reorgering logic to pytest_collection_finish hook, solves the problem, though it feels like a hacky workaround.

I see 3 potential causes the the issue in hand:

• I misunderstood the feature;
• pytest docs are wrong;
• pytest_collection_modifyitems has a regression.

I'd be grateful for the help with this one.

pip list:

Package                   Version         Editable project location
------------------------- --------------- ---------------------------
allure-pytest             2.13.2
allure-python-commons     2.13.2
ansicolors                1.1.8
attrs                     23.2.0
autobahn                  23.6.2
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
cryptography              42.0.5
hyperlink                 21.0.0
idna                      3.7
iniconfig                 2.0.0
isodate                   0.6.1
Jinja2                    3.1.3
jsonschema                4.21.1
jsonschema-path           0.3.2
jsonschema-specifications 2023.12.1
labgrid                   23.1a2
lazy-object-proxy         1.10.0
MarkupSafe                2.1.5
more-itertools            10.2.0
mypy                      1.8.0
mypy-extensions           1.0.0
openapi-core              0.19.0
openapi-schema-validator  0.6.2
openapi-spec-validator    0.7.1
packaging                 23.2
parse                     1.20.1
pathable                  0.4.3
pexpect                   4.9.0
pip                       24.0
pluggy                    1.4.0
ptyprocess                0.7.0
pycparser                 2.21
pyserial-labgrid          3.5.0.2
pytest                    8.0.0
pyudev                    0.24.1
pyusb                     1.2.1
PyYAML                    6.0.1
referencing               0.31.1
requests                  2.31.0
rfc3339-validator         0.1.4
rpds-py                   0.18.0
ruff                      0.2.1
setuptools                69.1.1
six                       1.16.0
txaio                     23.1.1
types-PyYAML              6.0.12.20240311
types-requests            2.31.0.20240311
typing_extensions         4.9.0
urllib3                   2.2.0
Werkzeug                  3.0.1
xmodem                    0.4.7

[bluetech]

It's possible some other plugin reorders as well? Try like this:

@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(config, items):
   ...

[github-actions[bot]]

This issue is stale because it has the status: needs information label and requested follow-up information was not provided for 14 days.

[github-actions[bot]]

This issue was closed because it has the status: needs information label and follow-up information has not been provided for 7 days since being marked as stale.