PEX_INHERIT_PATH+default shebangs don't interact well with virtualenvs

[benjyw]

Our default shebang is minor version-specific, e.g., /usr/bin/env python3.7. But a virtualenv doen't have a bin/python3.7, just bin/python and bin/python3. So even if running a pex in a virtualenv, that pex will bypass the virtualenv and run directly against the underlying interpreter.

Normally this is fine, since pex scrubs non-stdlib packages off the sys.path anyway. Where this becomes noticeable is when using PEX_INHERIT_PATH to attempt to get the pex to recognize the virtualenv's installed packages. The packages won't be visible, for reasons that aren't obvious to the user.

You can get this to work by changing the shebang (with --python-shebang="/usr/bin/env python3" for example), but that may be undesirable for other reasons.

At runtime you would expect to get this to work with PEX_PYTHON=python3 (or the full path to the venv interpreter), but for reasons unclear to me that doesn't work.

I don't see a solution, but wanted to document this gotcha. It's a subtle way in which pex ends up behaving differently from running raw python. Possibly we should add something about this to the pex docs.

When does this matter? When the pex in question is a tool that operates on python code. for example, say we build mypy.pex and run it directly on some code. mypy needs to introspect that code's dependencies at runtime, and a common way to do that is to run it in a venv. Pants of course takes care of all this for you, this is more about directly running a standalone mypy.pex from the cmdline.

[benjyw]

@jsirois any ideas or thoughts?

[benjyw]

Hmm, this might be more nuanced. In some cases I can see a bin/python3.7 in the virtualenv and in others not. Still trying to work through this a little.

[jsirois]

PEX_INHERIT_PATH behavior is non-obvious in general. The only really sane way to get what you mean is to say PEX_INHERIT_PATH=... /this/python my.pex .... That style says inherit /this/python's sys.path in the way I say for this PEX file. Probably a better way to spell the same thing would be using PEX_EXTRA_SYS_PATH pointing to the site-libraries directory of the interpreter you care about. This lacks fidelity with the the PEX_INHERIT_PATH=prefer variant though.

As to the venv only having python and python<minor> - that's surprising. I've not seen that.

[jsirois]

Re-reading the original description, I find PEX_PYTHON works just fine:

/tmp/pex-issues-1344 $ python -mvenv example
/tmp/pex-issues-1344 $ source example/bin/activate
(example) /tmp/pex-issues-1344 $ pip -q install -U pip
(example) /tmp/pex-issues-1344 $ pip -q install requests pex
(example) /tmp/pex-issues-1344 $ which pex
/tmp/pex-issues-1344/example/bin/pex
(example) /tmp/pex-issues-1344 $ pex -odefault_shebang_empty.pex
/tmp/pex-issues-1344 $ head -1 default_shebang_empty.pex 
#!/usr/bin/env python3.9
/tmp/pex-issues-1344 $ PEX_PYTHON=/tmp/pex-issues-1344/example/bin/python PEX_INHERIT_PATH=fallback ./default_shebang_empty.pex -c 'import requests'
/tmp/pex-issues-1344 $ PEX_PYTHON=/tmp/pex-issues-1344/example/bin/python ./default_shebang_empty.pex -c 'import requests'
Traceback (most recent call last):
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 484, in execute
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 543, in _execute
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 583, in execute_interpreter
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/pex.py", line 650, in execute_content
  File "/tmp/pex-issues-1344/default_shebang_empty.pex/.bootstrap/pex/compatibility.py", line 93, in exec_function
  File "-c <cmd>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

The only suggestion I have is adding more words to --inherit-path help, PEX_INHERIT_PATH help and in the relevant section of docs/buildingpex.rst. @benjyw if that sounds reasonable to you I can prepare a PR for that.

[benjyw]

Ah yes, clarifying that PEX_INHERIT_PATH is intended to work in tandem with PEX_PYTHON (at least if you want to inherit some interpreter's site-packages) would be great. Good call! Thanks.

[jsirois]

Ok. That's not really what I meant - I was just pointing out I did not repro your failure to get PEX_PYTHON working. In the docs I'll be proposing the simpler PEX_INHERIT_PATH=... /this/python my.pex ... I mentioned above.

[benjyw]

👍