"Import file mismatch" at test collection time

[lelit]

trafficstars

I tried to upgrade pytest from 8.0.2 to latest 8.1.1 and I got this error:

___________ ERROR collecting tests/conftest.py _____________________
import file mismatch:
imported module 'conftest' has this __file__ attribute:
  .../tests/bio/conftest.py
which is not the same as the test file we want to collect:
  .../tests/conftest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

Neither removing all __pycache__ subdirs, nor playing with the --import-mode option fixed it.

The top level tests directory contains this structure:

$ tree -d tests/
tests/
├── __pycache__
├── bio
│   └── __pycache__
├── models
│   └── __pycache__
├── scr
└── server
    └── __pycache__

and there are different conftest.py modules:

$ find tests/ -name 'conftest.py'
tests/conftest.py
tests/bio/conftest.py
tests/server/conftest.py

Given that I could not find a related breaking change alert, I'd say that the "use a unique basename for your test file modules" hint does not apply to conftest.py, right?

Thanks&bye.

Environment

I'm using Python 3.11.8 on a Debian sid, with the following packages in the virtual environment (after downgrading pytest to the working 8.0.2):

$ pip list
Package                         Version  Editable project location
------------------------------- -------- -------------------------
alabaster                       0.7.16
alembic                         1.13.1
attrs                           23.2.0
Babel                           2.14.0
beautifulsoup4                  4.12.3
bleach                          6.1.0
calmjs.parse                    1.3.1
certifi                         2024.2.2
cffi                            1.16.0
chardet                         5.2.0
charset-normalizer              3.3.2
commonmark                      0.9.1
coverage                        7.4.4
cryptography                    42.0.5
docutils                        0.20.1
flake8                          7.0.0
greenlet                        3.0.3
hupper                          1.12.1
idna                            3.6
imagesize                       1.4.1
importlib_metadata              7.0.2
iniconfig                       2.0.0
itsdangerous                    2.1.2
jaraco.classes                  3.2.3
jeepney                         0.8.0
Jinja2                          3.1.3
keyring                         24.3.1
Mako                            1.3.2
markdown-it-py                  2.2.0
MarkupSafe                      2.1.5
mccabe                          0.7.0
mdurl                           0.1.2
metapensiero.extjs.desktop      2.4
metapensiero.sqlalchemy.dbloady 3.0.dev3
metapensiero.sqlalchemy.proxy   6.0.dev5
metapensiero.tool.bump-version  1.3
more-itertools                  10.2.0
mypy                            1.6.1
mypy-extensions                 1.0.0
nh3                             0.2.14
packaging                       24.0
PasteDeploy                     3.1.0
pillow                          10.2.0
pip                             24.0
pkginfo                         1.10.0
plaster                         1.1.2
plaster-pastedeploy             1.0.1
pluggy                          1.4.0
ply                             3.11
progressbar2                    4.4.2
pycodestyle                     2.11.1
pycountry                       23.12.11
pycparser                       2.21
pyflakes                        3.2.0
Pygments                        2.17.2
PyNaCl                          1.5.0
pyparsing                       3.1.2
pyramid                         2.0.2
pyramid-mailer                  0.15.1
pyramid-mako                    1.1.0
pyramid-tm                      2.5
pyRXP                           3.0.1
pytest                          8.0.2
pytest-cov                      4.1.0
python-rapidjson                1.16
python-utils                    3.8.2
pytz                            2024.1
rcssmin                         1.1.2
readme_renderer                 43.0
reportlab                       4.1.0
repoze.sendmail                 4.4.1
requests                        2.31.0
requests-toolbelt               1.0.0
rfc3986                         2.0.0
rich                            13.7.1
rjsmin                          1.2.2
ruamel.yaml                     0.18.6
ruamel.yaml.clib                0.2.8
ruff                            0.1.1
SecretStorage                   3.3.3
setuptools                      69.2.0
six                             1.16.0
snowballstemmer                 2.2.0
SoL                             4.21     /home/lele/wip/sol5/src
soupsieve                       2.5
Sphinx                          7.2.6
sphinxcontrib-applehelp         1.0.8
sphinxcontrib-devhelp           1.0.6
sphinxcontrib-htmlhelp          2.0.5
sphinxcontrib-jsmath            1.0.1
sphinxcontrib-qthelp            1.0.7
sphinxcontrib-serializinghtml   1.1.10
SQLAlchemy                      2.0.28
tqdm                            4.66.2
transaction                     4.0
translationstring               1.4
twine                           5.0.0
typing_extensions               4.10.0
urllib3                         2.2.1
venusian                        3.1.0
Versio                          0.5.0
waitress                        3.0.0
wcwidth                         0.2.13
webencodings                    0.5.1
WebOb                           1.8.7
WebTest                         3.0.0
wheel                           0.43.0
XlsxWriter                      3.2.0
zipp                            3.18.1
zope.deprecation                5.0
zope.interface                  6.2
zope.sqlalchemy                 3.1

[nicoddemus]

Hi @lelit, thanks for the report.

I could not reproduce the issue however, here is my file tree:

└───tests
    │   conftest.py
    │
    ├───bio
    │   │   conftest.py
    │   │   tests_bio.py
    │
    ├───models
    │   │   test_models.py
    │
    ├───scr
    │   │   test_scr.py
    │
    ├───server
    │   │   conftest.py
    │   │   test_server.py   

(all files are empty)

This runs pytest file for me (with zero collected tests of course).

Can you try to provide a minimal working example?

[lelit]

Oh! I will try to distill an MWE, but for now a relatively quick (it takes a couple of minutes in total) recipe is:

$ docker run -ti python:3.11 bash
# cd /tmp
# git clone -b sol5 https://gitlab.com/metapensiero/SoL.git
# cd SoL
# python -m venv env
# source env/bin/activate
# pip install -r requirements/development.txt
# make test

The above works ok, producing

=========================== test session starts ===========================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /tmp/SoL
configfile: setup.cfg
plugins: cov-4.1.0
collected 304 items                                                       

src/sol/i18n.py .                                                   [  0%]
src/sol/models/utils.py ....                                        [  1%]
...
========= 299 passed, 5 skipped, 12 warnings in 66.61s (0:01:06) ==========

Then if I do

# pip install pytest==8.1.1
# make test

I get the following instead:

=========================== test session starts ===========================
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /tmp/SoL
configfile: setup.cfg
plugins: cov-4.1.0
collected 304 items / 1 error                                             

================================= ERRORS ==================================
___________________ ERROR collecting tests/conftest.py ____________________
import file mismatch:
imported module 'conftest' has this __file__ attribute:
  /tmp/SoL/tests/bio/conftest.py
which is not the same as the test file we want to collect:
  /tmp/SoL/tests/conftest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

As said, I will try to minimize it.

In the meantime, thanks for your time!

[xflr6]

Thanks! Same here: https://github.com/xflr6/graphviz/actions/runs/8317341967 (with multiple conftest.py files).

[mgorny]

If that's any help, I was able to workaround the problem by specifying --ignore tests/conftest.py --ignore tests/backend/conftest.py ....

[lelit]

Hi, a little update: as promised I tried to come up with an MWE that exhibits the problem, but so far I had no luck in isolating the condition that triggers it. In another, even bigger project I wrote and maintain it does not happen, and neither in a very simplicistic mock, similar to what @nicoddemus did above, but with not completely empty files (test units and conftest.py). I will try again.

[mgorny]

I suspect --doctest-modules may be relevant.

[lelit]

Thanks, I will take that into consideration, even if the other project I mentioned uses doctests as well but does not manifest the problem.

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity.

[alerque]

Lose the stale bot. This issue isn't stale as downstream stuff is still struggling with this and related issues that link back to this are not fixed. But not just this issue, lose the bot entirely.

[The-Compiler]

@alerque The stale bot only triggers on "status: needs information" issues, because very often people end up not providing any follow-up infos. The issue here is that nobody removed the label after that happened.

[lelit]

I'm sorry, I could not reserve some spare time to further investigate the issue, trying to condense a standalone replication recipe.

[lelit]

While I was not able to distill a repeatable MWE recipe, I executed a git bisect running pytest --collect-only in my project, basically doing

git bisect start
git bisect bad 8.2.0
git bisect good 8.0.2
git run sh run.sh

where run.sh is

pip install -U .
cd ~/wip/sol5/
pytest --collect-only

The final outcome is the following:

06dbd3c21ccdf1ac76e8fa264048133cb4660842 is the first bad commit
commit 06dbd3c21ccdf1ac76e8fa264048133cb4660842
Author: Ran Benita <[email protected]>
Date:   Sat Jan 13 11:15:05 2024 +0200

    doctest: remove special conftest handling
    
    (Diff better viewed ignoring whitespace)
    
    Since e1c66ab0ad8eda13e5552dfc939e07d7290ecd39, conftest loading is
    handled at the directory level before sub-nodes are collected, so there
    is no need for the doctest plugin to handle it specially.
    
    This was probably the case even before
    e1c66ab0ad8eda13e5552dfc939e07d7290ecd39, but I haven't verified this.

 src/_pytest/doctest.py | 26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)
bisect found first bad commit

Hope this brings some enlightenment.

[lelit]

That didn't work, or wasn't noticed?

[lelit]

FWIW, the issue is still present in v8.3.1, I'm a bit surprised nobody else is experiencing it, and no hint come up here. Oh well.

[nicoddemus]

Hi @lelit thanks for checking.

I am short on time myself to look into this more deeply however... some things to try meanwhile:

1. Provide a MWE, I know this is hard, but is much easier on us maintainers when a MWE is provided.
2. There was a question about --doctest-modules, perhaps remove that flag and see if the problem disappears? Not suggesting dropping it altogether, just to help find the root cause.
3. Are there any symlinks in that test suite, like a conftest.py which is sumlinked to another location?

[lelit]

Thank you! I will try harder to get an MWE, I failed with two distinct attempts in the past... But yes, I see that removing --doctest-modules from the addopts configuration in [tool.pytest.ini_options] the problem goes away. I will focus in that direction looking for an MWE. Note that I have several other projects using doctest, and they work ok with any version of pytest, so the problem is subtle.

[nicoddemus]

Another thing to look for is if there are symlinks in the repo, or if some file imports conftest files explicitly.

[RonnyPfannschmidt]

I believe adding __init__.py files in all folders might resolve this

[lelit]

Another thing to look for is if there are symlinks in the repo

Not in the Python sources, there is a single one between /docs/changes.rst and /CHANGES.rst.

or if some file imports conftest files explicitly.

No, git grep conftest does not detect any match.

I believe adding init.py files in all folders might resolve this

YES!

This

find tests -type d | grep -v __pycache__ | while read d; do touch $d/__init__.py; done

made a difference indeed: at first the errors bumped to six, all of them like the following:

make test
== test session starts ==
platform linux -- Python 3.12.4, pytest-8.3.1, pluggy-1.5.0
rootdir: /home/lele/wip/sol5
configfile: pyproject.toml
plugins: cov-5.0.0
collected 328 items / 6 errors                                                                                                                                                                


__ ERROR collecting tests/bio/test_serialization_gzip.py __
ImportError while importing test module '/home/lele/wip/sol5/tests/bio/test_serialization_gzip.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/bio/test_serialization_gzip.py:9: in <module>
    from test_serialization import full_dump_reload
E   ModuleNotFoundError: No module named 'test_serialization'
...
ERROR tests/bio/test_serialization_gzip.py
ERROR tests/bio/test_serialization_gzip.py
ERROR tests/bio/test_serialization_json.py
ERROR tests/bio/test_serialization_json.py
ERROR tests/bio/test_serialization_json_gzip.py
ERROR tests/bio/test_serialization_json_gzip.py
!! Interrupted: 6 errors during collection !!
== 1 warning, 6 errors in 6.83s ==

At that point the following patch

--- a/tests/bio/test_serialization_gzip.py
+++ b/tests/bio/test_serialization_gzip.py
@@ -6,7 +6,7 @@
 # :Copyright: © 2018 Lele Gaifax
 #
 
-from test_serialization import full_dump_reload
+from .test_serialization import full_dump_reload
 
 
 def test_full_dump_reload_gzip(session, tourney_rated, player_fata, player_lele):

on those tests made pytest (even latest 8.3.2) happy and working again on this repo!

:love_letter:

[nicoddemus]

Ahh great news, glad the problem was found @lelit!

I'm closing this then.

I wonder if we can improve pytest somehow to detect this problem, to at least give a warning?

[pushfoo]

TL;DR: RonnyPfannschmidt's solution works for me

I had the same issues and tried the same things as others in the thread:

• [x] addopts = ["--doctest-modules"] in my [tool.pytest.ini.options] section
• [x] Ruled out .pyc issues by running pyclean

This is what worked (from above):

I believe adding __init__.py files in all folders might resolve this

I'm on Python 3.10 with the following pip freeze output:

exceptiongroup==1.2.2
iniconfig==2.0.0
mypy==1.11.1
mypy-extensions==1.0.0
nodeenv==1.9.1
packaging==24.1
pluggy==1.5.0
pyclean==3.0.0
pyright==1.1.373
pytest==8.2.2
tomli==2.0.1
typing_extensions==4.12.2