uv-managed python installed into .local/bin can't be used with venv module

[bluss]

uv 0.4.30

python3.12 -m venv fails to create a virtual environment.

Steps how I reproduced this problem. Using 3.12.6 to use a version that was not already installed.

rm .local/bin/python3.12
uv python install --preview 3.12.6
# Installed Python 3.12.6 in 2.60s
#  + cpython-3.12.6-linux-x86_64-gnu (python3.12)
which python3.12
# /home/user/.local/bin/python3.12
python3.12 -V
# Python 3.12.6
python3.12 -m venv new_environment
# Error: Command '['/home/user/new_environment/bin/python3.12', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Let's try execute the python in the venv:

./new_environment/bin/python3.12

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './new_environment/bin/python3.12'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/user/.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/user/new_environment/bin/python3.12'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007136928b6740 (most recent call first):
  <no Python frame>

Is the symlink in the virtual environment resolved? (no)

ls -l ./new_environment/bin/python3.12 
# lrwxrwxrwx 1 user user 33 nov  5 03:08 ./new_environment/bin/python3.12 -> /home/user/.local/bin/python3.12*

[charliermarsh]

This is known and reported upstream: https://github.com/indygreg/python-build-standalone/issues/380

[zanieb]

See also #8429

[garyo]

I'm running into this; uv-installed python symlinked into $HOME/.local/bin fails to create a working venv. I've read #8429 above and https://github.com/indygreg/python-build-standalone/issues/380 as well. Is there any workaround at the user level that I can use? My specific use case is in a github workflow; one of the actions calls python -m venv .action-bin and I can't change that, but I can adjust anything else beforehand. I have uv 0.5.8, installed just now, into a CentOS 7 docker container.

[zanieb]

I think export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))" should work, though it's pretty hacky.

[garyo]

I think export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))" should work, though it's pretty hacky.

Sadly, that doesn't work:

[root@4b801b307a52 aswf]# curl -LsSf https://astral.sh/uv/install.sh | sh
System glibc version (`2.17') is too old; checking alternatives
downloading uv 0.5.8 x86_64-unknown-linux-musl-static
no checksums to verify
installing to /root/.local/bin
  uv
  uvx
everything's installed!

To add $HOME/.local/bin to your PATH, either restart your shell or run:

    source $HOME/.local/bin/env (sh, bash, zsh)
    source $HOME/.local/bin/env.fish (fish)
[root@4b801b307a52 aswf]# source ~/.local/bin/env
[root@4b801b307a52 aswf]# uv python install --preview 3.11
Installed Python 3.11.11 in 2.10s
 + cpython-3.11.11-linux-x86_64-gnu (python3.11)
[root@4b801b307a52 aswf]# (cd ~/.local/bin; ln -sf python3.11 python3; ln -sf python3.11 python)
[root@4b801b307a52 aswf]# which python
/root/.local/bin/python
[root@4b801b307a52 aswf]# python --version
Python 3.11.11
[root@4b801b307a52 aswf]# export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))"
[root@4b801b307a52 aswf]# echo $PYTHONHOME
/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu
[root@4b801b307a52 aswf]# python -mvenv foo
Error: Command '['/opt/aswf/foo/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

[zanieb]

Alas, if you use --without-pip it works fine. I'm... not sure why the ensurepip call fails there. Of course there's VIRTUALENV_WITHOUT_PIP but no venv equivalent to my knowledge.

Annoyingly, it works in isolation which makes it hard to inspect:

❯ python -m venv .venv
Error: Command '['/Users/zb/workspace/uv/.venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
 - pip==24.2
❯ .venv/bin/python -m ensurepip --upgrade --default-pip
Looking in links: /var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpgx6oozdb
Processing /var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpgx6oozdb/pip-24.2-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-24.2

[garyo]

I ended up prepending the actual python bin dir to $PATH. That seems to work for me.

Details at https://github.com/AcademySoftwareFoundation/openfx/commit/8276ac8156b55147da6343765cb1211d81c6ef23 if anyone’s interested.

[sgherdao]

this works for me

export PATH=$(dirname $(realpath $(which python))):$PATH

@garyo might be better than using ls -1 which may not work if you have multiple python interpreter

[treyhunner]

I launch Python (REPL, scripts, venv, etc.) with different versions of Python frequently when teaching and am trying to use uv as a pyenv-replacement (and much more) by using uv python install --default and I ran across this issue while trying to create virtual environments in uv-managed Python versions.

@sgherdao's PATH hack worked for me for my default python command.

Since I need this for all installed Python versions, I am using this:

# Add all uv-managed Python tool bins to PATH
uv python list --managed-python --only-installed --color=never \
  | awk '{print $2}' \
  | tac \
  | while read python_executable; do
    python_bin_dir="$(dirname "$python_executable")"
    case ":$PATH:" in
      *":$python_bin_dir:"*) :;; # Already in PATH
      *) PATH="$python_bin_dir:$PATH";; # Not yet in PATH
    esac
  done

The tac reverses the output (with the assumption that the newest Python versions come first and oldest last which seems to be the current output order of uv python list --managed-python --only-installed. This overwrites my global pip command and python command of course, but that's fine for my use case as I always alias python to the latest installed version.

This is a hack, but it seems to work for my use case for now.

[zanieb]

We're looking at fixing this.

[DIMFLIX]

Bypassing via PATH doesn't work for me :(

╭╴  dimflix at ~                                                                                                                                                                󱑈 11:54
╰─󰍟 uv python list --managed-python --only-installed
cpython-3.12.11-linux-x86_64-gnu    .local/share/uv/python/cpython-3.12-linux-x86_64-gnu/bin/python3.12
cpython-3.11.13-linux-x86_64-gnu    .local/share/uv/python/cpython-3.11-linux-x86_64-gnu/bin/python3.11
cpython-3.9.23-linux-x86_64-gnu     .local/share/uv/python/cpython-3.9.23-linux-x86_64-gnu/bin/python3.9

╭╴  dimflix at ~                                                                                                                                                                󱑈 11:57
╰─󰍟 echo $(dirname $(realpath $(which python)))
/home/dimflix/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/bin

╭╴  dimflix at ~                                                                                                                                                                󱑈 11:57
╰─󰍟 /home/dimflix/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/bin/python -m venv .venv
Error: Command '['/home/dimflix/.venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.