pypa
Using system-packages leads to module import error
I'm not able to import an installed dependency when running tests with system-packages. I have a ~convoluted setup with:
- A system python installation with site-packages (global, maintained by others)
- A hatch installation in a
venv(local, maintained by me) which points to the system python (include-system-site-packages = truein the pyvenv.cfg)
For example, I have two projects:
hatch new "Hatch Demo1"
hatch new "Hatch Demo2"
With an extra dependency not installed in the system python (python==3.7.3):
[tool.hatch.envs.default]
dependencies = [
"coverage[toml]>=6.5",
"pytest",
"impunity" # your package of choice
]
dev-mode = true
I use dev-mode in both and system-packages in hatch-demo1 but not hatch-demo2.
Running tests (hatch run test) results in an import error (ModuleNotFoundError: No module named 'impunity') for hatch-demo1 but not `hatch-demo2:
tests/test_import.py
def test_import_pytest():
import impunity
To make this more complicated, I can import impunity no problem using hatch run python to start an interpreter and importing impunity.
I've noticed that my sys.path is different between the failing/passing scenarious where the failing test environment has the following for sys.path (/opt/software/python is the global install):
>>> import sys
>>> sys.path
['~/software/git_repos/hatch-demo/hatch-demo1', '/home/shelf1/Software/python/3.7/bin', '/opt/Software/python/3.7/lib/python37.zip', '/opt/Software/python/3.7/lib/python3.7', '/opt/Software/python/3.7/lib/python3.7/lib-dynload', '~/.local/lib/python3.7/site-packages', '/opt/Software/python/3.7/lib/python3.7/site-packages']
But the environment that successfully imports impunity has the following, noting the path to the virtual environment.
>>> import sys
>>> sys.path
['', '/opt/Software/python/3.7/lib/python37.zip', '/opt/Software/python/3.7/lib/python3.7', '/opt/Software/python/3.7/lib/python3.7/lib-dynload', '~/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/hatch-demo1/lib/python3.7/site-packages', '~/software/git_repos/hatch-demo/hatch-demo1', '~/.local/lib/python3.7/site-packages', '/opt/Software/python/3.7/lib/python3.7/site-packages']
Does the system-packages option work for you? https://hatch.pypa.io/latest/plugins/environment/virtual/#options
@ofek That is the offending option, but I guess I can make that more clear. If I set system-packages to false, I'm able to correctly import the impunity (ie. non-system package), but when I specify system-packages = true, my tests fail (hatch run test).
I've included a parital pyproject.toml below with my default environment setup:
[tool.hatch.envs.default]
dependencies = [
"coverage[toml]>=6.5",
"pytest",
"impunity"
]
system-packages = false # setting to true causes import issues while running tests
[tool.hatch.envs.default.scripts]
test = "pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
"- coverage combine",
"coverage report",
]
cov = [
"test-cov",
"cov-report",
]
Do you mind running hatch env prune and then testing with the code on the master branch? There may have been fixes recently.
That solved one issue, but I'm now in another pickle since hatch dropped support for 3.7 (understandably so). I was hoping I could get away with pointing to the 3.7 binary on my system (using the python setting), but I've had similar issues as before. Pointing to 3.8 resulted in similar issues. I've listed my environment and steps to reproduce in python3.8 below.
Testing environment:
[tool.hatch.envs.test]
dependencies = [
"coverage[toml]>=6.5",
"pytest",
"impunity"
]
python = "/opt/Software/python/3.8/bin/python"
system-packages = true
[tool.hatch.envs.test.scripts]
env = "python -c 'import sys;print(sys.path);import pytest;import impunity'"
test = "pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
"- coverage combine",
"coverage report",
]
cov = [
"test-cov",
"cov-report",
]
Steps to reproduce
>>> source "/path/to/hatch/bin/activate"
>>> hatch run prune
>>> hatch run test:env
[
'',
'/opt/Software/python/3.8/lib/python38.zip',
'/opt/Software/python/3.8/lib/python3.8',
'/opt/Software/python/3.8/lib/python3.8/lib-dynload',
'/beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/test/lib/python3.8/site-packages',
'/beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1',
'/opt/Software/python/3.8/lib/python3.8/site-packages'
]
>>> hatch run test:test
/bin/sh: pytest: command not found
>>> python -m pytest
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.3.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-4.0.0
collected 1 item
tests/test_import.py F [100%]
=============================================================================== FAILURES ================================================================================
__________________________________________________________________________ test_import_pytest ___________________________________________________________________________
def test_import_pytest():
> import impunity
E ModuleNotFoundError: No module named 'impunity'
tests/test_import.py:4: ModuleNotFoundError
======================================================================== short test summary info ========================================================================
FAILED tests/test_import.py::test_import_pytest - ModuleNotFoundError: No module named 'impunity'
=========================================================================== 1 failed in 0.18s ===========================================================================
I added a matrix of python (3.7, 3.8, 3.11) and added just the binaries to my $PATH.
Running hatch run test:test
hatch run test:test
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
/bin/sh: pytest: command not found
Running hatch run test:python -m pytest
>>> hatch run test:python -m pytest
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.7.3, pytest-5.0.0, py-1.8.0, pluggy-0.12.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: arraydiff-0.3, astropy-header-0.1.2, remotedata-0.3.2, openfiles-0.5.0, doctestplus-0.11.1
collected 1 item
tests/test_import.py . [100%]
=========================================================================== warnings summary ============================================================================
tests/test_import.py::test_import_pytest
tests/test_import.py::test_import_pytest
/opt/Software/python/3.7/lib/python3.7/site-packages/pytest_remotedata/plugin.py:65: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(pytest.__version__) < LooseVersion("3.6"):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================= 1 passed, 2 warnings in 0.20 seconds ==================================================================
────────────────────────────────────────────────────────────────────────────── test.py3.8 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.8.16, pytest-7.3.1, pluggy-1.0.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-3.5.0
collected 1 item
tests/test_import.py . [100%]
=========================================================================== 1 passed in 0.05s ===========================================================================
────────────────────────────────────────────────────────────────────────────── test.py3.11 ──────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.0.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-3.5.0
collected 1 item
tests/test_import.py . [100%]
=========================================================================== 1 passed in 0.15s ===========================================================================
Bonus round
Setting test = "python -m pytest {args:tests}" works, but leads to library issues since impunity is compiled. Those errors are on my side, so this error is expected.
>>> hatch run test:test
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.7.9, pytest-7.4.3, pluggy-1.2.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
collected 1 item
tests/test_import.py F [100%]
=============================================================================== FAILURES ================================================================================
__________________________________________________________________________ test_import_pytest ___________________________________________________________________________
def test_import_pytest():
> import impunity
E ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/test.py3.7/lib/python3.7/site-packages/impunity.cpython-37m-x86_64-linux-gnu.so)
tests/test_import.py:4: ImportError
======================================================================== short test summary info ========================================================================
FAILED tests/test_import.py::test_import_pytest - ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/te...
=========================================================================== 1 failed in 0.43s ===========================================================================
@ofek I have a workaround for whatever is causing these issues. I was able to import everything successfully by combining the above solution with an absolute path to the system python. This is slightly unfortunate as I won't be able to use the matrix inputs for testing.
Successful import with python3.7
[tool.hatch.envs.test]
dependencies = [
"coverage[toml]>=6.5",
"pytest",
"impunity"
]
python = "/opt/Software/python/3.7/bin/python3.7"
system-packages = true
[tool.hatch.envs.test.scripts]
env = "python -c 'import sys;print(sys.path)'"
test = "python3.7 -m pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
"- coverage combine",
"coverage report",
]
cov = [
"test-cov",
"cov-report",
]
Error requiring full python name
Note that setting the test script to just python -m pytest {args:test} resulted in an error. It looks like python was stripped from the command.
cmd [1] | 3.7 -m pytest tests
/bin/sh: 3.7: command not found