Virtual environment imports are given precedence over project root

[jhonatan-lopes]

Bug report

What's wrong

I am trying to setup my Django app project with Tox, mypy and django-stubs using the settings from tests.settings. My package dir structure is the following:

myproject
├── example/
├── myapp/
├── tests/
├── manage.py
├── poetry.lock
├── tox.ini
└── pyproject.toml

Initially, I was using a settings.py file from my example project. On my pyproject.toml I had:

[tool.django-stubs]
django_settings_module = "example.settings"

This configuration ran without issues.

Later, I wanted to keep the tests settings separate from the example settings, so I added a settings.py file to the tests folder and edited the pyproject.toml for the following:

[tool.django-stubs]
django_settings_module = "tests.settings"

When running mypy on this case, I ran onto the following error:

Error constructing plugin instance of NewSemanalDjangoPlugin

Traceback (most recent call last):
  File "/home/user/dev/myproject/.tox/mypy/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 96, in main
  File "mypy/main.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 231, in _build
  File "mypy/build.py", line 478, in load_plugins
  File "mypy/build.py", line 456, in load_plugins_from_config
  File "/home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/mypy_django_plugin/main.py", line 70, in __init__
    self.django_context = DjangoContext(self.plugin_config.django_settings_module)
  File "/home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/mypy_django_plugin/django/context.py", line 96, in __init__
    apps, settings = initialize_django(self.django_settings_module)
  File "/home/jplopes/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/mypy_django_plugin/django/context.py", line 78, in initialize_django
    settings._setup()
  File "/home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/jplopes/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tests.settings'
ERROR: InvocationError for command /home/user/dev/myproject/.tox/mypy/bin/mypy test_houses (exited with code 1)

I believe that this is caused by mypy and django-stubs paths being configured first to myproject/.tox/mypy which finds a tests module which was pip installed from another dependency (alive-progress) in the .tox/mypy virtual environment before finding it on the project root. Since this left-over tests package does not have a settings.py file the error is thrown.

I used the following to monkey patch NewSemanalDjangoPlugin.__init__ from mypy_django_plugin.main to display the sys path location:

class NewSemanalDjangoPlugin(Plugin):
    def __init__(self, options: Options) -> None:
        super().__init__(options)
        self.plugin_config = DjangoPluginConfig(options.config_file)
        # Add paths from MYPYPATH env var
        sys.path.extend(mypy_path())
        # Add paths from mypy_path config option
        sys.path.extend(options.mypy_path)
        # Print `tests` package location
        import tests
        for idx, path in enumerate(sys.path, 1):
            print(f'{idx} - {path}')
        print(f'\ntests module location - {tests.__file__}')
        self.django_context = DjangoContext(self.plugin_config.django_settings_module)

which printed the following:

1 - /home/user/dev/myproject/.tox/mypy/bin
2 - /usr/lib/python38.zip
3 - /usr/lib/python3.8
4 - /usr/lib/python3.8/lib-dynload
5 - /home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages
6 - /home/user/dev/myproject

tests module location - /home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages/tests/__init__.py

How is that should be

I don't know if this was by design, but I believe that the project root should take precedence here to avoid these conflicts, i.e. the paths should be on the following order:

1 - /home/user/dev/myproject
2 - /home/user/dev/myproject/.tox/mypy/bin
3 - /usr/lib/python38.zip
4 - /usr/lib/python3.8
5 - /usr/lib/python3.8/lib-dynload
6 - /home/user/dev/myproject/.tox/mypy/lib/python3.8/site-packages

tests module location - /home/user/dev/myproject/tests/__init__.py

I believe that the expected behaviour with imports is that the project root takes precedence, which is not followed here. This has the potential to cause hard to trace bugs, such as this one.

System information

• OS: Linux-5.15.32-1-MANJARO-x86_64-with-glibc2.34
• python version: 3.8.13
• django version: 3.2.13
• mypy version: 0.942
• django-stubs version: 1.10.1
• django-stubs-ext version: 0.4.0