Error using pex for some packages using setuptools>=50.*

[g0di]

Hello folks,

Lastly I encountered an issue while deploying one of my application using pex caused by a weird import error for setuptools._distutils. I could reproduce the issue and discovered it is caused when using setuptools>=50.* during the pex build for some python packages (not all)

Steps to reproduce (Python3.5, pex2.1)

python -m virtualenv .venv
.venv/bin/pip install pex~=2.1 setuptools>=50.*
.venv/bin/pex -v flask-jwt -o test.pex

Executing previous shell script, you will end up with following error

(.venv) /tmp/test-gody# .venv/bin/pex -v flask-jwt -o test.pex
pex: Building pex :: Resolving distributions (['flask-jwt']) :: Resolving for:
pex: Hashing pex                                                                                                                                                                                               pex: Hashing pex: 47.6ms                                       
pex: Isolating pex: 0.2ms
pex: PEX.run invoking /tmp/test-gody/.venv/bin/python3 /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /root/.pex download --dest /tmp/tmpu28y_b3l/tmp.test-gody..venv.bin.python3 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 flask-jwt
pex: Found site-library: /tmp/test-gody/.venv/lib/python3.5/site-packages
pex: Tainted path element: /tmp/test-gody/.venv/lib/python3.5/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.5/site-packages
pex: Scrubbing from site-packages: /tmp/test-gody/.venv/lib/python3.5/site-packages
pex: New sys.path: ['/root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.bootstrap', '/root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610', '/tmp/test-gody/.venv/lib/python35.zip', '/tmp/test-gody/.venv/lib/python3.5', '/tmp/test-gody/.venv/lib/python3.5/plat-x86_64-linux-gnu', '/tmp/test-gody/.venv/lib/python3.5/lib-dynload', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu']
pex: Activating PEX virtual environment from /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610: 13.3ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610
pex:   * /tmp/test-gody/.venv/lib/python35.zip
pex:     /tmp/test-gody/.venv/lib/python3.5
pex:     /tmp/test-gody/.venv/lib/python3.5/plat-x86_64-linux-gnu
pex:     /tmp/test-gody/.venv/lib/python3.5/lib-dynload
pex:     /usr/lib/python3.5
pex:     /usr/lib/python3.5/plat-x86_64-linux-gnu
pex:     /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.deps/pip
pex:     /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.deps/setuptools
pex:     /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.deps/wheel
pex:     /root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
    ERROR: Command errored out with exit status 1:
     command: /tmp/test-gody/.venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-download-3erasexa/flask-jwt/setup.py'"'"'; __file__='"'"'/tmp/pip-download-3erasexa/flask-jwt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /tmp/pip-download-3erasexa/flask-jwt/pip-egg-info
         cwd: /tmp/pip-download-3erasexa/flask-jwt/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/root/.pex/pip.pex/2f9d8679bc44d45e25ce08ea3a84c7fd425a5610/.deps/setuptools/setuptools/__init__.py", line 6, in <module>
        import distutils.core
      File "/tmp/test-gody/.venv/lib/python3.5/site-packages/_distutils_hack/__init__.py", line 83, in create_module
        return importlib.import_module('setuptools._distutils')
      File "/tmp/test-gody/.venv/lib/python3.5/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ImportError: No module named 'setuptools._distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

If you change setuptools to a version before 50 it works as expected

.venv/bin/pip install -U "setuptools<50.*"
.venv/bin/pex -v flask-jwt -o test.pex

If you try to install flask instead of flask-jwt it works and I have no idea why

Note that if you install flask-jwt manually like .venv/bin/pip install flask-jwt it works fine as well

[novemberkilo]

We've experienced this too. Suggest it is related to https://github.com/pypa/setuptools/issues/2353

There appears to be a fix in v50.1.0 https://setuptools.readthedocs.io/en/latest/history.html

We bumped our version of setuptools to v50.3.0 and can confirm that we are no longer seeing this issue.

[jsirois]

Thanks @novemberkilo! It looks like a workaround then for setuptools 50 is to export SETUPTOOLS_USE_DISTUTILS=stdlib. That is all the 50.1.0 fix ~does. I'm pretty sure it doesn't make sense to release a patch to Pex to set this env var since setuptools folks appear to be actively trying to migrate to SETUPTOOLS_USE_DISTUTILS=local.

[jsirois]

This problem no-longer reproduces with modern Pex:

:; python3.5 -mvenv .venv

:; .venv/bin/pip --cert /etc/ssl/certs/ca-certificates.crt install pex~=2.1 setuptools==50
Collecting pex~=2.1
  Using cached https://files.pythonhosted.org/packages/de/45/94497d22a1517b2462394f641ea272e7ec624823f223c01a5f0d7e6f571d/pex-2.16.2-py2.py3-none-any.whl
Collecting setuptools==50
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/b0/8b/379494d7dbd3854aa7b85b216cb0af54edcb7fce7d086ba3e35522a713cf/setuptools-50.0.0-py3-none-any.whl
Installing collected packages: pex, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed pex-2.16.2 setuptools-50.0.0
You are using pip version 9.0.1, however version 24.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

:; .venv/bin/pex -V
2.16.2

:; .venv/bin/pex --disable-cache -v flask-jwt -o test.pex
pex: Hashing pex: 80.8ms
pex: Isolating pex: 42.8ms
pex:   Extracting pex to /tmp/tmpsrjlraqk/isolated/024db4f0f215687d5dd48690ea17704c7d39650c: 42.1ms
pex: Building pex :: Resolving distributions for requirements: flask-jwt :: Resolving requirements. :: Resolving for:
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/activate

pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/activate.csh
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/activate.fish
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/easy_install
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/easy_install-3.8
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/pip
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/pip3
pex: Re-writing /tmp/tmpsrjlraqk/venvs/a1a0a2cea735cc36f072dcdb4f5a64cf915a4142/7aff4d0838cb9a4a402485f1f883a8fc85cc2bcd.lck.work/bin/pip3.8
pex: Building pex :: Resolving distributions for requirements: flask-jwt :: Resolving requirements. :: Building distributions for:
  BuildRequest(target=LocalInterpreter(id='home.jsirois.dev.pex-tool.pex..venv.bin.python3.5', platform=Platform(platform='manylinux_2_39_x86_64', impl='cp', version='3.5.10', version_info=(3, 5, 10), abi='cp35m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.5.10', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='5.15.153.1-microsoft-standard-WSL2', platform_system='Linux', platform_version='#1 SMP Fri Mar 29 23:14:13 UTC 2024', python_full_version='3.5.10', python_version='3.5', sys_platform='linux'), interpreter=PythonInterpreter('/home/jsirois/dev/pex-tool/pex/.venv/bin/python3.5', PythonIdentity('/home/jsirois/dev/pex-tool/pex/.venv/bin/python3.5', 'cp35', 'cp35m', 'manylinux_2_39_x86_64', (3, 5, 10)))), source_path='/tmp/tmpsrjlraqk/downloads/resolver_download.xh1w4nh9/home.jsirois.dev.pex-tool.pex..venv.bin.python3.5/Flask-JWT-0.3.2.tar.gz', fingerprint='49c0672fbde0f1cd3pex: Building /tmp/tmpsrjlraqk/downloads/resolver_download.xh1w4nh9/home.jsirois.dev.pex-tool.pex..venv.bin.python3.5/Flask-JWT-0.3.2.tar.gz to /tmp/tmpsrjlraqk/built_wheels/sdists/Flask-JWT-0.3.2.tar.gz/49c0672fbde0f1cd3374bd834918d28956e3c521c7e00089cdc5380d323bd0ad/cp35-cp35m-manylinux_2_39_x86_64




pex: Building pex :: Resolving distributions for requirements: flask-jwt :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='flask-jwt', processed_text='flask-jwt', source='<string>', start_line=1, end_line=1), requirement=Requirement(name='flask-jwt', specifier=<SpecifierSet(''pex: Building pex :: Resolving distributions for requirements: flask-jwt :: Resolving requirements. :: Installing 8 distributions

pex: Elapsed time per install job:tributions for requirements: flask-jwt :: Resolving requirements. :: Installing 8 distributions :: Using 2 parallel jobs to install 8 items
  1) [341221] 0.04s 4 wheels
  2) [341220] 0.04s 4 wheels
pex: Building pex: 14529.2ms
pex:   Adding distributions from pexes: : 0.1ms
pex:   Adding distributions built from local projects and collecting their requirements: : 0.7ms
pex:     Resolving requirements.: 0.4ms
pex:   Resolving distributions for requirements: flask-jwt: 14526.4ms
pex:     Resolving requirements.: 14526.3ms
pex:       Resolving for:
  cp35-cp35m-manylinux_2_39_x86_64 interpreter at /home/jsirois/dev/pex-tool/pex/.venv/bin/python3.5: 12174.2ms
pex:       Building distributions for:
  BuildRequest(target=LocalInterpreter(id='home.jsirois.dev.pex-tool.pex..venv.bin.python3.5', platform=Platform(platform='manylinux_2_39_x86_64', impl='cp', version='3.5.10', version_info=(3, 5, 10), abi='cp35m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.5.10', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='5.15.153.1-microsoft-standard-WSL2', platform_system='Linux', platform_version='#1 SMP Fri Mar 29 23:14:13 UTC 2024', python_full_version='3.5.10', python_version='3.5', sys_platform='linux'), interpreter=PythonInterpreter('/home/jsirois/dev/pex-tool/pex/.venv/bin/python3.5', PythonIdentity('/home/jsirois/dev/pex-tool/pex/.venv/bin/python3.5', 'cp35', 'cp35m', 'manylinux_2_39_x86_64', (3, 5, 10)))), source_path='/tmp/tmpsrjlraqk/downloads/resolver_download.xh1w4nh9/home.jsirois.dev.pex-tool.pex..venv.bin.python3.5/Flask-JWT-0.3.2.tar.gz', fingerprint='49c0672fbde0f1cd3374bd834918d28956e3c521c7e00089cdc5380d323bd0ad'): 2210.8ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='flask-jwt', processed_text='flask-jwt', source='<string>', start_line=1, end_line=1), requirement=Requirement(name='flask-jwt', specifier=<SpecifierSet('')>, marker=None, url=None, extras=frozenset()), editable=False): 0.1ms
pex:       Installing 8 distributions: 108.6ms
pex:         Using 2 parallel jobs to install 8 items: 108.1ms
pex:       Checking install: 1.4ms
pex:   Configuring PEX dependencies: 1.0ms
Saving PEX file to test.pex
pex: Zipping PEX file.: 108.8ms

The resolved distribution for flask-jwt is Flask-JWT-0.3.2.tar.gz which was released on November 3rd, 2015 which must be the same version you were resolving. As such I'll close this issue.