bin directory of python packages no longer in PATH

[mdorier]

Steps to reproduce

If you install a python package, the resulting files will be located in <prefix>/local instead of <prefix>. While this doesn't seem to be a problem for importing the module, it is a problem when the python package installs executables in bin. These executables end up in <prefix>/local/bin, which in not in PATH when loading the package or when in an environment that contains the package.

To reproduce, simply try installing a python package that has executables (e.g. py-flake8), then try to use said executable.

Error message

No response

Information on your system

• Spack: 0.19.0.dev0 (123354e920582607af689db83037c1055e4ff778)
• Python: 3.10.4
• Platform: linux-ubuntu22.04-sandybridge
• Concretizer: clingo

General information

• [X] I have run spack debug report and reported the version of Spack/Python/Platform
• [X] I have searched the issues of this repo and believe this is not a duplicate
• [X] I have run the failing commands in debug mode and reported the output

[adamjstewart]

This was also reported in #31939. It seems to be due to distro-specific patches to the system Python. Only current workaround is to use Spack-installed Python instead of system Python.

[adamjstewart]

@trws @pradyunsg

[mdorier]

Any chance spack could just add local/bin in addition to bin in PATH?

[adamjstewart]

That would break a lot of external packages. For example, imagine you have Python 3.7 in /usr/bin/python and Python 3.9 in /usr/local/bin/python. If you use:

- spec: python
  prefix: /usr

which will it pick up?

[tmadlener]

After just running into this (with meson in my case); Is there any news on this or on #31939? Or is this currently waiting for the upstream situation to resolve somehow?

[adamjstewart]

It looks like there was some recent discussion on this in https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/27 by @pradyunsg and @FFY00, not sure if they have anything else to add.

[FFY00]

No, not yet unfortunately, but shouldn't users be using a Spack-provided Python?

[adamjstewart]

Spack supports using external installations of all packages. A lot of users who don't care about Python and just need it as a dependency for their HPC code will use an external copy of Python instead of letting Spack built it from source to save time. We have a spack external find command that automatically searches the OS and picks up as many system-installed packages as it can.

[adamjstewart]

For anyone who encounters this issue, see https://github.com/pypa/pip/issues/10978 for the upstream tracking issue. For now, the best workaround is to use a Spack-installed Python instead of using the system Python.

This issue has been reported several times already:

• #31939
• #37632
• #33923
• #32769

I've closed these issues so we can track this issue in only a single place.