pytest-django could not find a Django project

[ptrhck]

trafficstars

Thanks to this example repo, I started experimenting with Django. I have added a simple django project following this tutorial while using the same dependency versions as in this repo. My pants.toml looks like this:

[GLOBAL]
pants_version = "2.11.0"

backend_packages = [
  "pants.backend.python",
]

[anonymous-telemetry]
enabled = false


[python]
# We use a narrow interpreter constraint to ensure that Python PEX'es will be executable by our
# selected Docker base image, `python:3.10`.
interpreter_constraints = ["==3.9.*"]
enable_resolves = true
resolves = { python-default = "lockfiles/python-default.txt" }
lockfile_generator = "pex"

[source]
root_patterns = [
  "/django"
]

[pytest]
lockfile = "lockfiles/pytest.txt"
version = "pytest>=6.2.4,<6.3"
extra_requirements.add = [
  "pytest-django>=4,<5",
]

[python-infer]
# Infer dependencies from strings that look like module/class names, such as are often
# found in settings.py, where dependencies are enumerated as strings and not directly imported.
string_imports = true
string_imports_min_dots = 1

My project strucuture looks like this

pants.toml
pytest.ini
  /django
    manage.py
    /mysite
      settings.py
    /polls
      tests.py

My pytest.ini looks as follows:

[pytest]
DJANGO_SETTINGS_MODULE = mysite.settings
pythonpath = .

I have been experimenting with the pythonpath option, but I am not able to run the test with ./pants test django/polls/tests.py as it throws the following error:

15:17:29.99 [INFO] Initializing scheduler...
15:17:30.13 [INFO] Scheduler initialized.
15:17:31.63 [WARN] Failed to generate JUnit XML data for django/polls/tests.py:tests.
15:17:31.63 [ERROR] Completed: Run Pytest - django/polls/tests.py:tests failed (exit code 1).
Traceback (most recent call last):
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pytest_django/plugin.py", line 179, in _handle_import_error
    yield
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pytest_django/plugin.py", line 351, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mysite'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/process-execution6KqLW2/.cache/pex_root/venvs/0b6a9b078e047f3c7341f09b2e9163fb0d6a7996/83912a035d4805bd1b6618bfd9e98f78cda4d615/pex", line 182, in <module>
    sys.exit(func())
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1191, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pytest_django/plugin.py", line 351, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/vscode/.cache/pants/named_caches/pex_root/venvs/s/64a59a10/venv/lib/python3.9/site-packages/pytest_django/plugin.py", line 183, in _handle_import_error
    raise ImportError(msg)
ImportError: No module named 'mysite'

Any idea? Is the pytest PYTHONPATH messing with the pants PYTHONPATH? I also thought, that I would not need to specify the path as it is a standard Django installation with a default location for manage.py.

You can find the source code here.

[ptrhck]

I figured out the following so far:

1. pythonpath is only available with pytest>=7. Installing all packages and tools with just pip and running pytest from the root directory as outlined above works with this pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE = mysite.settings
python_files = tests.py test_*.py *_tests.py
pythonpath = django

vscode ➜ /workspaces/example-docker-django $ pip freeze | grep 'Django\|pytest'
Django==3.2.13
pytest==7.1.2
pytest-django==4.5.2
vscode ➜ /workspaces/example-docker-django $ pytest
================================================================================================================================= test session starts ==================================================================================================================================
platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0
django: settings: mysite.settings (from ini)
rootdir: /workspaces/example-docker-django, configfile: pytest.ini
plugins: django-4.5.2
collected 2 items                                                                                                                                                                                                                                                                      

django/polls/tests.py F.                                                                                                                                                                                                                                                         [100%]

======================================================================================================================================= FAILURES =======================================================================================================================================
_____________________________________________________________________________________________________________________ YourTestClass.test_something_that_will_fail ______________________________________________________________________________________________________________________

self = <tests.YourTestClass testMethod=test_something_that_will_fail>

    def test_something_that_will_fail(self):
>       self.assertTrue(False)
E       AssertionError: False is not true

django/polls/tests.py:16: AssertionError
=============================================================================================================================== short test summary info ================================================================================================================================
FAILED django/polls/tests.py::YourTestClass::test_something_that_will_fail - AssertionError: False is not true
============================================================================================================================= 1 failed, 1 passed in 0.24s ==============================================================================================================================

2. Running ./pants test :: still throws the above error. If I set pythonpath = /workspaces/example-docker-django/django (absolute path), the following is thrown:

django.core.exceptions.ImproperlyConfigured: The app module <module 'polls' (namespace)> has multiple filesystem locations (['/tmp/process-executionvZSHuN/django/polls', '/workspaces/example-docker-django/django/polls']); you must configure this app with an AppConfig subclass with a 'path' class attribute.

If I add the path attribute to apps.py it works:

from django.apps import AppConfig
import os
from django.conf import settings

class PollsConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'polls'
    path = os.path.join(settings.BASE_DIR, 'polls')

Why? I cannot image that this is the right setup.

[kaos]

To follow up from Slack thread.

This is due to there are no inferrable dependency from the polls test to the mysite.settings module, so that is not included in the sandbox running the tests.

Adding this dependency explicitly in the BUILD file should resolve this issue:

diff --git a/django/polls/BUILD b/django/polls/BUILD
index 0eea8b1..72520e3 100644
--- a/django/polls/BUILD
+++ b/django/polls/BUILD
@@ -2,4 +2,5 @@ python_sources()
 
 python_tests(
     name="tests",
+    dependencies=["django/mysite/settings.py"],
 )

[ptrhck]

That does the trick. As discussed in the Slack thread I tried the new __defaults__ feature (https://github.com/pantsbuild/pants/pull/15836, https://github.com/pantsbuild/pants/pull/15923) to solve this. But this does not work as the mysite module cannot be found again. Source code can be found here https://github.com/ptrhck/example-django (Note the PANTS_SHA in .devcontainer.json).

Also, ./pants run django:manage -- runserver is not working as it results in:

  File "/workspaces/example-docker-django/.pants.d/tmp2njknb9j/django/manage.py", line 10, in main
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspaces/example-docker-django/.pants.d/tmp2njknb9j/django/manage.py", line 21, in <module>
    main()
  File "/workspaces/example-docker-django/.pants.d/tmp2njknb9j/django/manage.py", line 12, in main
    raise ImportError(
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?

However, ./pants run django:manage -- migrate works. Any idea? As this is a very common Django project structure, this could be of help for others.

[benjyw]

Does ./pants run django:manage -- runserver --noreload work?

Django's dev server does some funky re-exec stuff in reload mode that I think is defeated by ./pants run, at least when in sandbox mode (Pants 2.13 has an option to run directly from in-repo sources, which may work better).

And in any case, when sandboxed, Django's autoreloading won't work. But (assuming --noreload fixes things) you can set restartable=True on the django:manage pex_binary target and let Pants do the autoreloading for you. It's more precise than Django's anyway.

[ptrhck]

Thanks for the explanation, that worked!

@kaos any idea for the defaults? I was sure you had a working examples on Slack, but this config does not work as shown above.

[kaos]

@ptrhck what is not working? From your example repo, I think it seems to work, from what I can tell, at least.

$ PANTS_SHA=7382ad47b42618b155d8c3a0ea63b3827bdca574 ./pants dependencies django/polls/tests.py 
23:28:33.99 [INFO] Initializing scheduler...
23:28:34.22 [INFO] Scheduler initialized.
//:root#django
django/mysite/settings.py

When I comment out the default

diff --git a/django/BUILD b/django/BUILD
index e7b4224..f64bed4 100644
--- a/django/BUILD
+++ b/django/BUILD
@@ -6,5 +6,5 @@ pex_binary(
 )
 
 __defaults__({
-      (python_tests): dict(dependencies=["django/mysite/settings.py"])
-    })
\ No newline at end of file
+#      (python_tests): dict(dependencies=["django/mysite/settings.py"])
+})

I get just:

$ PANTS_SHA=7382ad47b42618b155d8c3a0ea63b3827bdca574 ./pants dependencies django/polls/tests.py 
//:root#django

Or is it another issue?

Edit: just noticed you had another commit in your test repo. Seems you changed from python_tests to python_test for the default, which will then not apply defaults to python_tests targets in your BUILD files. That those generate python_test targets doesn't matter, as the defaults are not applied "through" the python_tests target generator, so you must target your defaults for the specific targets used in the BUILD file. Hope that makes sense.