Use `/usr/local/bin/python3.<minor_version>` if available on macOS as default python interpreter

[itsayellow]

trafficstars

How would this feature be useful?

See #633, and others. On macOS with Homebrew, our use of sys.executable to find the default (non-user-specified) python interpreter gives a more specific file location different than /usr/local/bin/python3 (e.g. /usr/local/opt/[email protected]/bin/python3.9). When this python executable file is used to create a venv, it links to a VERY specific python executable full path that changes with each sub-sub-minor upgrade by Homebrew of its python3 (e.g. /usr/local/Cellar/[email protected]/3.9.2_1/bin/python3.9)

The upshot of this is that every single time Homebrew ever upgrades its python (more often than new Python sub-minor versions), all of a macOS user's pipx venvs break, necessitating a run of pipx reinstall-all. This can happen multiple times a month, or even multiple times a week (see https://github.com/Homebrew/homebrew-core/commits/578501f1556b739170edf4c991ca12a49ac82616/Formula/python%403.9.rb)

See below on macOS. pipx uses sys.executable as in the example below to find a python interpreter to use to create a venv:

> /usr/local/bin/python3
Python 3.9.2 (default, Feb 24 2021, 13:26:09) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/[email protected]/bin/python3.9'

Then using that to create a venv:

> /usr/local/opt/[email protected]/bin/python3.9 -m venv test_venv
> ls -l test_venv/bin/python3.9
[...] test_venv/bin/python3.9@ -> /usr/local/Cellar/[email protected]/3.9.2_1/bin/python3.9

This last ultra-specific path is what breaks pipx venvs when Homebrew upgrades its python. (Before Homebrew version 3.9.2_1 there were 3.91, 3.91_1, 3.91_2, 3.91_3, 3.91_4, 3.91_5, 3.91_6, 3.91_7, 3.91_8)

Describe the solution you'd like

If we notice that we are on macOS, and we find that /usr/local/bin/python3.<minor_version> exists, we should use THAT as the default python interpreter on macOS.

In this case we get longer-lived valid symlinks in each venv

> /usr/local/bin/python3.9 -m venv test_venv
> ls -l test_venv/bin/python3.9       
[...] test_venv/bin/python3.9@ -> /usr/local/bin/python3.9

Describe alternatives you've considered

Otherwise we keep forcing our macOS users (like me) to keep executing pipx reinstall-all for ever sub-sub-minor Homebrew python upgrade.

[uranusjr]

There are a couple of issues I can think of:

1. /usr/local is not necessarily the correct prefix, since Homebrew can be installed in alternative locations (HOMEBREW_PREFIX environment variable).
2. /usr/local/bin/pythonX.Y is also the installation target with the python.org macOS Python distribution (IIRC), so it may not point to the Homebrew Python even if it exists.

I think we should come up with a full interpreter discovery logic on macOS instead, similar to pipx.interpreter._find_default_windows_interpreter(), instead of trying to fix sys.executable in special cases.

[gaborbernat]

I think we should come up with a full interpreter discovery logic on macOS instead, similar to pipx.interpreter._find_default_windows_interpreter(), instead of trying to fix sys.executable in special cases.

You could also just use https://github.com/pypa/virtualenv/tree/main/src/virtualenv/discovery 🤔 rather than reimplement it

[uranusjr]

I love virtualenv.discovery, but virtualenv is a big thing to pull in. I would reach for it every time if the functionality could be extracted into its own distribution.

[gaborbernat]

The only reason we haven't done that is that no one complained yet that it's too big 😆

[itsayellow]

Well, virtualenv is also MIT license, so one option would just be to pull out the virtualenv.discovery code and "vendor" it in pipx.

[gaborbernat]

I don't think you want to fix upcoming bugs 🤐

[itsayellow]

How would one use virtualenv.discovery to locate a good python interpreter? I was poking around but couldn't figure out the way in.

Would it find help on macOS to find e.g. /usr/local/bin/python3 instead of /usr/local/opt/[email protected]/bin/python3.9?

[uranusjr]

I believe virtualenv.discovery mainly searches PATH on POSIX systems, so it would find /usr/local/bin/python3 without issues, and likely won’t be able to see /usr/local/opt/[email protected]/bin/python3.9 at all.

Another thing, would it be also useful to change the sys.executable call to sys._base_executable?

[itsayellow]

Another thing, would it be also useful to change the sys.executable call to sys._base_executable?

I was wondering about that and forgot to mention it. On macOS sys._base_executable actually returns the right thing /usr/local/bin/python3.

I was a little worried that it was both undocumented and with the starting underscore indicating internal use.

If we felt confident in using it, I think it would help a lot, at least on macOS.

[uranusjr]

I think we should ask first, although I’m not sure where.

[gaborbernat]

I'm fairly certain @zooba told me at some point that can be used safely (especially if we make it something like getattr(sys, '_base_executable', fallback)

[itsayellow]

Hmmm, it looks like the shebang at the top of the Homebrew-installed pipx script is:

#!/usr/local/opt/[email protected]/bin/python3.9

So inside of pipx code installed by homebrew, sys._base_executable returns /usr/local/Cellar/[email protected]/3.9.2_1/bin/python3.9, which is the same old problem. 😝

[itsayellow]

It looks like on non-Windows systems at least, searching the PATH might be the only answer.

[itsayellow]

@gaborbernat how do I code python discovery using virtualenv.discovery? I just want to try it out. I see things like PythonInfo.discover_exe that look promising, but I wasn't sure how to use it, e.g. what it wants for app_data.

[gaborbernat]

You want get_interpreter from here https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/builtin.py#L60:

from virtualenv.discovery.builtin import get_interpreter

result = get_interpreter('py', [])
print(result)
result.system

PythonInfo(spec=CPython3.9.2.final.0-64, system=/usr/local/opt/[email protected]/bin/python3.9, exe=/Users/bernat/git/github/virtualenv/.tox/4/dev/bin/python, platform=darwin, version='3.9.2 (default, Feb 24 2021, 13:30:36) \n[Clang 12.0.0 (clang-1200.0.32.29)]', encoding_fs_io=utf-8-utf-8)

Note this might not work exactly you'd want, so we might need to add one or two tweak knobs in there. The app data is there for cache purposes, if you don't want to cache you can leave it None.

[itsayellow]

Somehow this is no longer a problem on macOS with Homebrew. I couldn't find a modification to the [email protected] Homebrew formula that would seem to fix it, but now the venv python is no longer linked to the super-specific Cellar python path as in my first post on this bug.

> python3
Python 3.9.2 (default, Mar 15 2021, 17:37:51) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/[email protected]/bin/python3.9'
>>> exit()
> /usr/local/opt/[email protected]/bin/python3.9 -m venv test_venv
> ls -l test_venv/bin/python3.9 
lrwxr-xr-x  1 mclapp  staff  24 Feb 28 23:20 test_venv/bin/python3.9@ -> /usr/local/bin/python3.9

[itsayellow]

Also, now sys.executable seems to be "stable", i.e. once you use path /usr/local/opt/[email protected]/bin/python3.9 sys.executable resolves to that same path (it didn't seem to before.)

> /usr/local/opt/[email protected]/bin/python3.9                  
Python 3.9.2 (default, Mar 15 2021, 17:37:51) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/[email protected]/bin/python3.9'