tox
tox copied to clipboard
Published
20 hours ago •
tox-dev
tox-dev
referenced deps from an environment defined by an optional combinator are not pulled in
Issue
Suppose we have two environments defined by the below:
[testenv:lint{,-ci}]
deps =
flake8
flake8-print
flake8-black
ci: flake8-junit-report
commands =
!ci: flake8
ci: flake8 --output-file flake8.txt --exit-zero
ci: flake8_junit flake8.txt flake8_junit.xml
When we run these environments explicitly, the dependencies of each is appropriately handled -- i.e.
tox r -e lint --> installs flake8, flake8-print, flake8-black
tox r -e lint-ci --> installs flake8, flake8-print, flake8-black, flake8-junit-report
However, when we refer to these dependencies in another environment, such as with:
[testenv:safety]
deps =
{[tox]requires}
{[testenv]deps}
{[testenv:lint-ci]deps}
safety
commands =
safety check
Only the deps not prefixed with ci are pulled into the safety env when ran. This is regardless of how we refer to it -- {[testenv:lint]deps} also does not pull in the flake8-junit-report dependency.
Workaround seems to be to additionally define the optional combinator in the calling environment, i.e. define [testenv:safety{,-ci}] , then when we run tox r -e safety-ci -- it will pull in the ci prefix definitions in the referenced dependencies (but conversely, those defined with !ci would not be pulled in, even if we use the reference {[testenv:lint]deps}).
This workaround is undesirable as there is intended to only be one such "safety" environment, and expectation is that by providing the correct environment name in the reference ({[testenv:lint-ci]deps}) that it will pull the resources defined by the reference and not by those of the calling test env.
Environment
Provide at least:
- OS: Ubuntu18, 20
- tox: 3.21, 3.28, 4.6.2
Output of pip list of the host Python, where tox is installed
(safety) ubuntu:~/GitHub/NewTemp/CommonPy$ pip list
Package Version
------------------ ---------
aiohttp 3.8.4
aiosignal 1.3.1
async-timeout 4.0.2
attrs 23.1.0
bandit 1.7.5
black 22.12.0
boto3 1.20.11
botocore 1.23.11
certifi 2023.5.7
cffi 1.15.1
charset-normalizer 3.1.0
click 8.1.3
cryptography 41.0.1
distlib 0.3.6
docker 4.4.4
dparse 0.6.2
exceptiongroup 1.1.1
filelock 3.12.2
flake8 6.0.0
flake8-black 0.3.6
flake8-isort 6.0.0
flake8-print 5.0.0
frozenlist 1.3.3
gitdb 4.0.10
GitPython 3.1.31
idna 3.4
iniconfig 2.0.0
isort 5.12.0
Jinja2 3.1.2
jinja2-cli 0.8.2
jmespath 0.10.0
markdown-it-py 3.0.0
MarkupSafe 2.1.3
marshmallow 3.14.1
mccabe 0.7.0
mdurl 0.1.2
moto 2.2.16
multidict 6.0.4
mypy-extensions 1.0.0
packaging 21.3
pathspec 0.11.1
pbr 5.11.1
pip 22.0.4
pipdeptree 2.9.3
platformdirs 3.6.0
pluggy 1.0.0
psycopg2-binary 2.9.2
py 1.11.0
pycodestyle 2.10.0
pycparser 2.21
pyflakes 3.0.1
Pygments 2.15.1
PyMySQL 1.0.2
pyparsing 3.1.0
pytest 7.3.2
python-dateutil 2.8.2
pytz 2023.3
PyYAML 5.4.1
requests 2.31.0
responses 0.23.1
rich 13.4.2
ruamel.yaml 0.17.32
ruamel.yaml.clib 0.2.7
s3transfer 0.5.2
safety 2.3.5
setuptools 68.0.0
six 1.16.0
slackclient 2.9.3
smmap 5.0.0
sort-requirements 1.3.0
stevedore 5.1.0
toml 0.10.2
tomli 2.0.1
tox 3.28.0
tox-docker 1.7.0
tox-venv 0.4.0
types-PyYAML 6.0.12.10
typing_extensions 4.6.3
urllib3 1.26.16
virtualenv 20.23.1
websocket-client 1.6.0
Werkzeug 2.3.6
wheel 0.40.0
xmltodict 0.13.0
yarl 1.9.2
Output of running tox
Output of tox -rvv
ubuntu:~/GitHub/NewTemp/CommonPy$ tox r -vve safety
using tox.ini: /home/ubuntu/GitHub/NewTemp/CommonPy/tox.ini (pid 27634)
removing /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/log
python3.8 (/usr/bin/python3.8) is {'executable': '/usr/bin/python3.8', 'implementation': 'CPython', 'version_info': [3, 8, 16, 'final', 0], 'version': '3.8.16 (default, Dec 7 2022, 01:12:13) \n[GCC 7.5.0]', 'is_64': True, 'sysplatform': 'linux', 'os_sep': '/', 'extra_version_info': None}
safety uses /usr/bin/python3.8
unit uses /usr/bin/python3.8
coverage uses /usr/bin/python3.8
build uses /usr/bin/python3.8
dev uses /usr/bin/python3.8
release uses /usr/bin/python3.8
using tox-3.28.0 from /home/ubuntu/.local/lib/python3.6/site-packages/tox/__init__.py (pid 27634)
skipping sdist step
safety start: getenv /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety
safety reusing: /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety
safety finish: getenv /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety after 0.03 seconds
safety start: finishvenv
safety finish: finishvenv after 0.01 seconds
safety start: envreport
setting PATH=/home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin:/home/ubuntu/GitHub/.vscode-server/bin/252e5463d60e63238250799aef7375787f68b4ee/bin/remote-cli:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[27757] /home/ubuntu/GitHub/NewTemp/CommonPy$ /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin/python -m pip freeze >.tox/safety/log/safety-3.log
safety finish: envreport after 0.42 seconds
safety installed: bandit==1.7.5,black==22.12.0,commonpy==3.2.2,certifi==2023.5.7,charset-normalizer==3.1.0,click==8.1.3,distlib==0.3.6,docker==4.4.4,dparse==0.6.2,exceptiongroup==1.1.1,filelock==3.12.2,flake8==6.0.0,flake8-black==0.3.6,flake8-isort==6.0.0,flake8-print==5.0.0,gitdb==4.0.10,GitPython==3.1.31,idna==3.4,iniconfig==2.0.0,isort==5.12.0,Jinja2==3.1.2,jinja2-cli==0.8.2,markdown-it-py==3.0.0,MarkupSafe==2.1.3,mccabe==0.7.0,mdurl==0.1.2,mypy-extensions==1.0.0,packaging==21.3,pathspec==0.11.1,pbr==5.11.1,pipdeptree==2.9.3,platformdirs==3.6.0,pluggy==1.0.0,py==1.11.0,pycodestyle==2.10.0,pyflakes==3.0.1,Pygments==2.15.1,pyparsing==3.1.0,pytest==7.3.2,PyYAML==6.0,requests==2.31.0,rich==13.4.2,ruamel.yaml==0.17.32,ruamel.yaml.clib==0.2.7,safety==2.3.5,six==1.16.0,smmap==5.0.0,sort-requirements==1.3.0,stevedore==5.1.0,toml==0.10.2,tomli==2.0.1,tox==3.28.0,tox-docker==1.7.0,tox-venv==0.4.0,typing_extensions==4.6.3,urllib3==1.26.16,virtualenv==20.23.1,websocket-client==1.6.0
removing /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/tmp
safety start: run-test-pre
safety run-test-pre: PYTHONHASHSEED='2925510828'
safety run-test-pre: commands[0] | /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin/python -m pip install -r requirements.txt
setting PATH=/home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin:/home/ubuntu/GitHub/.vscode-server/bin/252e5463d60e63238250799aef7375787f68b4ee/bin/remote-cli:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[27759] /home/ubuntu/GitHub/NewTemp/CommonPy$ /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin/python -m pip install -r requirements.txt
...
Installing collected packages: types-PyYAML, pytz, xmltodict, werkzeug, PyYAML, python-dateutil, PyMySQL, pycparser, psycopg2-binary, multidict, marshmallow, jmespath, frozenlist, attrs, async-timeout, yarl, responses, cffi, botocore, aiosignal, s3transfer, cryptography, aiohttp, slackclient, boto3, moto
Attempting uninstall: PyYAML
Found existing installation: PyYAML 6.0
Uninstalling PyYAML-6.0:
Successfully uninstalled PyYAML-6.0
Successfully installed PyMySQL-1.0.2 PyYAML-5.4.1 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 boto3-1.20.11 botocore-1.23.11 cffi-1.15.1 cryptography-41.0.1 frozenlist-1.3.3 jmespath-0.10.0 marshmallow-3.14.1 moto-2.2.16 multidict-6.0.4 psycopg2-binary-2.9.2 pycparser-2.21 python-dateutil-2.8.2 pytz-2023.3 responses-0.23.1 s3transfer-0.5.2 slackclient-2.9.3 types-PyYAML-6.0.12.10 werkzeug-2.3.6 xmltodict-0.13.0 yarl-1.9.2
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety/bin/python -m pip install --upgrade pip' command.
safety finish: run-test-pre after 21.28 seconds
safety start: run-test
safety run-test: commands[0]....
Minimal example
As defined above along with some others in the tox "requires" section (shouldn't be impactful, but if needed can provide full tox.ini), we can see flake8-junit-report is not listed to be installed:
tox -e safety
safety create: /home/ubuntu/GitHub/NewTemp/CommonPy/.tox/safety
safety installdeps: urllib3<2, tox>=3.21.0,<4, tox-venv, tox-docker<2, setuptools>=65.5.1, wheel, pip>=21.3.1, flake8, flake8-print, flake8-black, flake8, flake8-isort, sort-requirements, bandit, setuptools>=65.5.1, wheel, pip>=21.3.1, pytest, jinja2-cli[yaml], pipdeptree, safety
