pypa
Trouble with packaging tutorial (solved by upgrading virtualenv)
- What is your operating system and version?
Ubuntu 18.04.5
- What is your Python version?
Python 3.8.2
- What version of pip do you have?
Pip 21.1.3
- If following an online tutorial or guide, please provide a link to the page or section giving you trouble:
https://packaging.python.org/tutorials/packaging-projects/
- Could you describe your issue in as much detail as possible?
I was following the tutorial I mentioned above, and when I got to the python3 -m build step I was hit with this stacktrace:
Traceback (most recent call last):
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/__main__.py", line 302, in main
build_call(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependency_check)
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/__main__.py", line 164, in build_package_via_sdist
sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/__main__.py", line 101, in _build
return _build_in_isolated_env(builder, outdir, distribution, config_settings)
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/__main__.py", line 77, in _build_in_isolated_env
with IsolatedEnvBuilder() as env:
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/env.py", line 92, in __enter__
executable, scripts_dir = _create_isolated_env_venv(self._path)
File "/home/gcaton/var/lib/virtualenvs/tmp-a7a86bb4e75279/lib/python3.8/site-packages/build/env.py", line 221, in _create_isolated_env_venv
pip_distribution = next(iter(metadata.distributions(name='pip', path=[purelib])))
StopIteration
ERROR
After some flailing, I realized that my virtualenv executable was the system-wide one at /usr/bin/virtualenv, which is pretty stale at version 15.1.0. I installed a newer version and it fixed all of my issues:
(tmp-a7a86bb4e75279) # 13:04:03 dev wip/packaging_tutorial> python3 -m pip install --upgrade virtualenv
Collecting virtualenv
Using cached virtualenv-20.4.7-py2.py3-none-any.whl (7.2 MB)
Collecting distlib<1,>=0.3.1
Using cached distlib-0.3.2-py2.py3-none-any.whl (338 kB)
Collecting appdirs<2,>=1.4.3
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting filelock<4,>=3.0.0
Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting six<2,>=1.9.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, filelock, distlib, appdirs, virtualenv
Successfully installed appdirs-1.4.4 distlib-0.3.2 filelock-3.0.12 six-1.16.0 virtualenv-20.4.7
Your tutorial recommends upgrading pip to the latest version - maybe it should include a similar note for virtualenv? (I'm using a clunky old virtualenvwrapper-based workflow to create temp envs, so I could certainly be the true problem here.)
Anyway, I had no problems after this - thanks for writing solid documentation.
It doesn't require virtualenv, actually. But if it has one ~~with a sufficient version~~, it should use that instead of venv. My guess is it's not adding pip to the virtualenv? Actually, now that I look at it, there's no version check. @gaborbernat, is there a minimum and should we add that to the check?
pip install build[virtualenv] will force virtualenv>=20.0.35. Should we ignore import virtualenv if it's older than 20.3.35?
The version check is here:
https://github.com/pypa/build/blob/5a3c5243a2135236f207da3a4b6401e1561f952d/src/build/env.py#L75
I thought we did, I just looked at the top and didn't see it. Also, the line that is breaking here is in the venv code, if I'd looked more carefully. I think we've seen this a time or two before, where pip seems to be missing from the installed venv. I'd love to be able to reproduce that in a container.
This has been observed on CentOS but not on Ubuntu - I thought Ubuntu threw an error when python3-venv or python3-pip weren't installed.
@votedmost What does a fresh venv look like on your system? (see the linked issue, could you do the same sort of check?) Do you have an idea of how you set it up? I see you have a newer pip than ubuntu provides?
@henryiii - my setup is an oddball. I downloaded and compiled Python 3.8.2 (from python.org) and installed it to ~/opt/Python-3.8.2. I used this interpreter to create a "default virtualenv" which is activated whenever I open a shell. This venv contains virtualenv 16.7.10. A few years ago I was stuck on Ubuntu 14.04 for various reasons (which include laziness) and developed this setup in order to get a newer version of Python 3 than was available in the Canonical repos at the time.
One wrinkle in this setup is that I use the system virtualenv executable to create that "default virtualenv". I hadn't realized up until this tutorial that all of my venvs also use that.
The user in the issue you linked also seems to be using virtualenvwrapper - we're both using mktmpenv to create a temporary venv, as you can see in their path /home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/. I also saw a warning about "ensurepip" being missing while I flailed around a bit.
I managed to repro this in an ubuntu:18.04 docker container. Here's my shell history, which I can clean up a bit later. (It takes a long time because it compiles Python and also installs a bunch of dependencies that might not be necessary - I just copy-pasted from a dotfiles repo I use.)
Well, as far as MREs go, this is a fairly long one. Can you repro it without installing python, etc., from apt? Can you repro it without virtualenvwrapper? Is virtualenvwrapper using your Python or the system Python?
I've looked a bit further into this one and found that the issue is (probably) not with build. venv, when invoked inside an existing virtualenv, produces a pretty sparse environment that inherits packages from the parent virtualenv using a mechanism I haven't seen before, pyvenv.cfg:
# 15:48:56 dev ~> python3 -m virtualenv /tmp/virtualenv
Using real prefix '/home/gcaton/opt/Python-3.8.2'
New python executable in /tmp/virtualenv/bin/python3
Also creating executable in /tmp/virtualenv/bin/python
Installing setuptools, pip, wheel...
done.
# 11:16:03 dev ~> source /tmp/virtualenv/bin/activate
(virtualenv) # 11:16:14 dev ~> python3 -m venv /tmp/venv/
(virtualenv) # 11:16:36 dev ~> tree /tmp/venv/
/tmp/venv/
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── python -> python3
│ └── python3 -> /tmp/virtualenv/bin/python3
├── include
├── lib
│ └── python3.8
│ └── site-packages
├── lib64 -> lib
└── pyvenv.cfg
6 directories, 7 files
# 11:17:52 dev ~> cat /tmp/venv/pyvenv.cfg
home = /tmp/virtualenv/bin
include-system-site-packages = false
version = 3.8.2
# 11:17:57 dev ~> source /tmp/venv/bin/activate
(venv) # 11:19:03 dev ~> python
Python 3.8.2 (default, Oct 30 2020, 11:57:27)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/tmp/virtualenv/lib/python38.zip', '/tmp/virtualenv/lib/python3.8', '/tmp/virtualenv/lib/python3.8/lib-dynload', '/home/gcaton/opt/Python-3.8.2/lib/python3.8', '/tmp/virtualenv/lib/python3.8/site-packages']
>>>
The stacktrace that I included is because build doesn't inspect the "parent" virtualenv for dependencies using this pyvenv.cfg mechanism.
When venv is invoked outside of a virtualenv, it copies all of the dependencies into the fresh environment:
# 11:30:19 dev ~> which python3
/usr/bin/python3
# 11:30:21 dev ~> python3 -m venv /tmp/venv2
# 11:30:28 dev ~> tree /tmp/venv2
/tmp/venv2
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── easy_install
│ ├── easy_install-3.6
│ ├── pip
│ ├── pip3
│ ├── pip3.6
│ ├── python -> python3
│ └── python3 -> /usr/bin/python3
├── include
├── lib
│ └── python3.6
│ └── site-packages
│ ├── easy_install.py
│ ├── pip
│ │ ├── basecommand.py
...
It does create the pyvenv.cfg file but it doesn't seem to munge sys.path in the same way:
# 11:33:07 dev ~> source /tmp/venv2/bin/activate
(venv2) # 11:33:10 dev ~> python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/tmp/venv2/lib/python3.6/site-packages']
>>>
This feels more like a bug in venv than build, but I'm not an expert here.
This appears to be an issue either with your custom Python installation or the interaction between virtualenv and your custom Python installation. Can you run python3 -m venv, with your Python, outside of a virtualenv? Does it still not contain pip?
@layday - I agree. My second example above uses the system-wide Python interpreter to show that there's no problem with a venv when created outside of a virtualenv, and the behavior is also correct when I use my custom installation outside of a virtualenv:
# 14:22:33 dev ~> # Nothing up my sleeves:
# 14:22:35 dev ~> env | grep -e VIRT -e PY
# 14:23:09 dev ~> /home/gcaton/opt/Python-3.8.2/bin/python3 -m venv /tmp/venv3
# 14:23:58 dev ~> tree /tmp/venv3
/tmp/venv3
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── easy_install
│ ├── easy_install-3.8
│ ├── pip
│ ├── pip3
│ ├── pip3.8
│ ├── python -> python3
│ └── python3 -> /home/gcaton/opt/Python-3.8.2/bin/python3
├── include
├── lib
│ └── python3.8
│ └── site-packages
│ ├── easy_install.py
│ ├── pip
│ │ ├── __init__.py
│ │ ├── _internal
│ │ │ ├── build_env.py
│ │ │ ├── cache.py
... snip ...