django
Instructions for running tox with docker don't work
As mentionned by Pat on Discord (not sure what their github username is), our current instructions for running tox with docker don't seem to work:
Full log
``` $ docker compose exec web tox py312-tests: install_deps> python -I -m pip install -r /usr/src/app/requirements/tests.txt Collecting django-push@ git+https://github.com/brutasse/django-push.git@22fda99641cfbd2f3075a723d92652a8e38220a5 (from -r /usr/src/app/requirements/common.txt (line 6)) Cloning https://github.com/brutasse/django-push.git (to revision 22fda99641cfbd2f3075a723d92652a8e38220a5) to /tmp/pip-install-xbk4s6l8/django-push_869cd076d70d473da4c86b60e8cecf7d Resolved https://github.com/brutasse/django-push.git to commit 22fda99641cfbd2f3075a723d92652a8e38220a5 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting Babel==2.16.0 (from -r /usr/src/app/requirements/common.txt (line 1)) Downloading babel-2.16.0-py3-none-any.whl.metadata (1.5 kB) Collecting django-contact-form==5.1.1 (from -r /usr/src/app/requirements/common.txt (line 2)) Downloading django_contact_form-5.1.1-py3-none-any.whl.metadata (2.7 kB) Collecting django-countries==7.6.1 (from -r /usr/src/app/requirements/common.txt (line 3)) Downloading django_countries-7.6.1-py3-none-any.whl.metadata (31 kB) Collecting django-hosts==5.1 (from -r /usr/src/app/requirements/common.txt (line 4)) Downloading django_hosts-5.1-py3-none-any.whl.metadata (4.7 kB) Collecting django-money==3.5.3 (from -r /usr/src/app/requirements/common.txt (line 5)) Downloading django_money-3.5.3-py3-none-any.whl.metadata (18 kB) Collecting django-read-only==1.18.0 (from -r /usr/src/app/requirements/common.txt (line 7)) Downloading django_read_only-1.18.0-py3-none-any.whl.metadata (7.1 kB) Collecting django-recaptcha==4.0.0 (from -r /usr/src/app/requirements/common.txt (line 8)) Downloading django_recaptcha-4.0.0-py3-none-any.whl.metadata (13 kB) Collecting django-registration-redux==2.13 (from -r /usr/src/app/requirements/common.txt (line 9)) Downloading django_registration_redux-2.13-py2.py3-none-any.whl.metadata (5.1 kB) Collecting Django==5.0.9 (from -r /usr/src/app/requirements/common.txt (line 10)) Downloading Django-5.0.9-py3-none-any.whl.metadata (4.1 kB) Collecting docutils==0.21.2 (from -r /usr/src/app/requirements/common.txt (line 11)) Downloading docutils-0.21.2-py3-none-any.whl.metadata (2.8 kB) Collecting feedparser==6.0.11 (from -r /usr/src/app/requirements/common.txt (line 12)) Downloading feedparser-6.0.11-py3-none-any.whl.metadata (2.4 kB) Collecting Jinja2==3.1.4 (from -r /usr/src/app/requirements/common.txt (line 13)) Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB) Collecting libsass==0.23.0 (from -r /usr/src/app/requirements/common.txt (line 14)) Downloading libsass-0.23.0-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (4.2 kB) Collecting Markdown==3.7 (from -r /usr/src/app/requirements/common.txt (line 15)) Downloading Markdown-3.7-py3-none-any.whl.metadata (7.0 kB) Collecting Pillow==11.0.0 (from -r /usr/src/app/requirements/common.txt (line 16)) Downloading pillow-11.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (9.1 kB) Collecting psycopg==3.2.3 (from psycopg[c]==3.2.3->-r /usr/src/app/requirements/common.txt (line 17)) Downloading psycopg-3.2.3-py3-none-any.whl.metadata (4.3 kB) Collecting Pygments==2.18.0 (from -r /usr/src/app/requirements/common.txt (line 18)) Downloading pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB) Collecting pykismet3==0.1.1 (from -r /usr/src/app/requirements/common.txt (line 19)) Downloading pykismet3-0.1.1-py2.py3-none-any.whl.metadata (2.9 kB) Collecting requests==2.32.3 (from -r /usr/src/app/requirements/common.txt (line 20)) Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Collecting sorl-thumbnail==12.11.0 (from -r /usr/src/app/requirements/common.txt (line 21)) Downloading sorl_thumbnail-12.11.0-py3-none-any.whl.metadata (10 kB) Collecting Sphinx==8.1.3 (from -r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinx-8.1.3-py3-none-any.whl.metadata (6.4 kB) Collecting stripe==3.1.0 (from -r /usr/src/app/requirements/common.txt (line 23)) Downloading stripe-3.1.0-py2.py3-none-any.whl.metadata (2.6 kB) Collecting time-machine==2.16.0 (from -r /usr/src/app/requirements/common.txt (line 24)) Downloading time_machine-2.16.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting django-debug-toolbar==4.4.6 (from -r /usr/src/app/requirements/dev.txt (line 2)) Downloading django_debug_toolbar-4.4.6-py3-none-any.whl.metadata (3.9 kB) Collecting isort==5.13.2 (from -r /usr/src/app/requirements/dev.txt (line 3)) Downloading isort-5.13.2-py3-none-any.whl.metadata (12 kB) Collecting pre-commit~=4.0.1 (from -r /usr/src/app/requirements/dev.txt (line 4)) Downloading pre_commit-4.0.1-py2.py3-none-any.whl.metadata (1.3 kB) Collecting watchdog==6.0.0 (from -r /usr/src/app/requirements/dev.txt (line 5)) Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl.metadata (44 kB) Collecting coverage==7.6.9 (from -r /usr/src/app/requirements/tests.txt (line 2)) Downloading coverage-7.6.9-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.2 kB) Collecting requests-mock==1.12.1 (from -r /usr/src/app/requirements/tests.txt (line 3)) Downloading requests_mock-1.12.1-py2.py3-none-any.whl.metadata (4.1 kB) Collecting tox-r /usr/src/app/requirements/common.txt (line 3)) Downloading asgiref-3.8.1-py3-none-any.whl.metadata (9.3 kB) Collecting typing-extensions (from django-countries==7.6.1->-r /usr/src/app/requirements/common.txt (line 3)) Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) Collecting setuptools (from django-money==3.5.3->-r /usr/src/app/requirements/common.txt (line 5)) Using cached setuptools-75.6.0-py3-none-any.whl.metadata (6.7 kB) Collecting py-moneyed=2.0 (from django-money==3.5.3->-r /usr/src/app/requirements/common.txt (line 5)) Downloading py_moneyed-3.0-py3-none-any.whl.metadata (3.2 kB) Collecting sqlparse>=0.3.1 (from Django==5.0.9->-r /usr/src/app/requirements/common.txt (line 10)) Downloading sqlparse-0.5.3-py3-none-any.whl.metadata (3.9 kB) Collecting sgmllib3k (from feedparser==6.0.11->-r /usr/src/app/requirements/common.txt (line 12)) Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting MarkupSafe>=2.0 (from Jinja2==3.1.4->-r /usr/src/app/requirements/common.txt (line 13)) Downloading MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Collecting charset-normalizer=2 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading charset_normalizer-3.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB) Collecting idna=2.5 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading idna-3.10-py3-none-any.whl.metadata (10 kB) Collecting urllib3=1.21.1 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB) Collecting certifi>=2017.4.17 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB) Collecting sphinxcontrib-applehelp>=1.0.7 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_applehelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-devhelp>=1.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_devhelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-htmlhelp>=2.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-jsmath>=1.0.1 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl.metadata (1.4 kB) Collecting sphinxcontrib-qthelp>=1.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_qthelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-serializinghtml>=1.1.9 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl.metadata (2.4 kB) Collecting snowballstemmer>=2.2 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading snowballstemmer-2.2.0-py2.py3-none-any.whl.metadata (6.5 kB) Collecting alabaster>=0.7.14 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading alabaster-1.0.0-py3-none-any.whl.metadata (2.8 kB) Collecting imagesize>=1.3 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading imagesize-1.4.1-py2.py3-none-any.whl.metadata (1.5 kB) Collecting packaging>=23.0 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB) Collecting python-dateutil (from time-machine==2.16.0->-r /usr/src/app/requirements/common.txt (line 24)) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting psycopg-c==3.2.3 (from psycopg[c]==3.2.3->-r /usr/src/app/requirements/common.txt (line 17)) Downloading psycopg_c-3.2.3.tar.gz (598 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 598.9/598.9 kB 3.2 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... errorRunning command git clone --filter=blob:none --quiet https://github.com/brutasse/django-push.git /tmp/pip-install-xbk4s6l8/django-push_869cd076d70d473da4c86b60e8cecf7d Running command git rev-parse -q --verify 'sha^22fda99641cfbd2f3075a723d92652a8e38220a5' Running command git fetch -q https://github.com/brutasse/django-push.git 22fda99641cfbd2f3075a723d92652a8e38220a5 error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [8 lines of output] running dist_info creating /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info writing /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/PKG-INFO writing dependency_links to /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/dependency_links.txt writing top-level names to /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/top_level.txt writing manifest file '/tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/SOURCES.txt' couldn't run 'pg_config' --includedir: [Errno 2] No such file or directory: 'pg_config' error: [Errno 2] No such file or directory: 'pg_config' [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed
× Encountered error while generating package metadata. ╰─> See above for output.
note: This is an issue with the package mentioned above, not pip. hint: See above for details. py312-tests: exit 1 (21.41 seconds) /usr/src/app> python -I -m pip install -r /usr/src/app/requirements/tests.txt pid=28 py312-tests: FAIL ✖ in 21.53 seconds py312-flake8: commands[0]> flake8 py312-flake8: OK ✔ in 2.4 seconds py312-black: OK ✔ in 0.02 seconds py312-isort: commands[0]> make isort-check python -m isort --check accounts aggregator blog contact dashboard djangoproject docs foundation fundraising legacy members releases svntogit tracdb py312-tests: FAIL code 1 (21.53 seconds) py312-flake8: OK (2.39=setup[0.01]+cmd[2.38] seconds) py312-black: OK (0.01 seconds) py312-isort: OK (0.87=setup[0.01]+cmd[0.85] seconds) evaluation failed :( (24.98 seconds)
</details
The [edit: first part of the] issue occurs due to tox attempting to install requirements in isolation, but the docker build process purging required packages.
https://github.com/django/djangoproject.com/blob/75f3fdee0bcc49b9502a2dd5d1d0221978c79091/Dockerfile#L37-L41
This creates a smaller image, which is good for production but does affect development.
I think the most elegant solution would be to use Docker's multi-stage builds, with base, dev, and production targets. Package installation could then be done in the base image, while unnecessary packages could be purged in production.
Tox setup removed from the project in https://github.com/django/djangoproject.com/pull/2135
