`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one

[redsun82]

A detailed description of the bug or problem you are having

When creating the following structure:

x/
├─ y/
│  ├─ y/
│  ├─ test_y.py

then pytest --import-mode=importlib fails with the following cryptic message:

================================= ERRORS =================================
_____________________ ERROR collecting x/y/test_y.py _____________________
<frozen importlib._bootstrap_external>:1448: in find_spec
    ???
<frozen importlib._bootstrap_external>:1222: in __init__
    ???
<frozen importlib._bootstrap_external>:1238: in _get_parent_path
    ???
E   KeyError: 'x'

To trigger this bug:

• the directory containing the test needs to be within at least another directory, it can't be directly underneath the root path
• the directory containing the test must in turn contain another directory with the same base name

Running with --pdb gives a slightly more useful backtrace:

  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(341)from_call()
-> result: Optional[TResult] = func()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(389)collect()
-> return list(collector.collect())
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(548)collect()
-> self._register_setup_module_fixture()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(561)_register_setup_module_fixture()
-> self.obj, ("setUpModule", "setup_module")
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(287)obj()
-> self._obj = obj = self._getobj()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(545)_getobj()
-> return importtestmodule(self.path, self.config)
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(492)importtestmodule()
-> mod = import_path(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(565)import_path()
-> mod = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(660)_import_module_using_spec()
-> parent_module = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(633)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
  <frozen importlib._bootstrap_external>(1448)find_spec()
  <frozen importlib._bootstrap_external>(1222)__init__()
> <frozen importlib._bootstrap_external>(1238)_get_parent_path()

Output of pip list from the virtual environment you are using

I tried this on a minimal poetry env with just pytest

-------------- -------
exceptiongroup 1.2.1
iniconfig      2.0.0
packaging      24.1
pip            24.0
pluggy         1.5.0
pytest         8.2.2
setuptools     70.0.0
tomli          2.0.1

pytest and operating system versions

pytest 8.2.2. Encountered this both with python 3.10 and python 3.12, so the python version doesn't seem to be a factor.

Running on Ubuntu 22.04.4 LTS.

cc @criemen who encountered this

[Zac-HD]

Feels like this might be an upstream issue; we should try to get a reproducer using stdlib code only.

[valeriupredoi]

cheers for fixing this, folks! I noticed a side-effect of this though: if pytest is running a test on a module that imports e.g.

from cow.moo.farm import udder

and there is an /cow/moo/moo.py, so module moo.py is in a dir identically called moo and there is no __init__.py in /moo then the test fails with an ImportError - not a biggie since IMHO since one uses package-level imports, then there must be init files all over the place where modules live, it's all good, but just wanted to let you know if yous get some others complaining about it :beer:

[michael-christen]

Seeing a similar issue as https://github.com/pytest-dev/pytest/issues/12592#issuecomment-2512039696

Perhaps it's my misunderstanding of namespace packages, but I typically don't have any __init__.py files in my repo. And I'm seeing an AttributeError for my module now:

 AttributeError: module 'third_party.my_protobuf_utils.my_protobuf_utils' has no attribute '_read_varint'

[dongfangtianyu]

Hi, @valeriupredoi @michael-christen Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

---

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

[michael-christen]

Hi, @valeriupredoi @michael-christen Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

Yep, that's the exact case I have, thanks @dongfangtianyu :pray: