pex scrubbing of sys.path and site-packages.

[pgroudas]

Hello,

It seems that in src/python/twitter/common/python/pex.py you're trying to eliminate all the extraneous load paths such that running a pex will not pick up externally installed dependencies.

  @classmethod
  def minimum_path(cls):
    """
      Return as a tuple the emulated sys.path and sys.path_importer_cache of
      a bare python installation, a la python -S.
    """
    site_libs = set(cls._site_libs())
    for site_lib in site_libs:
      TRACER.log('Found site-library: %s' % site_lib)
    for extras_path in cls._extras_paths():
      TRACER.log('Found site extra: %s' % extras_path)
      site_libs.add(extras_path)
    site_libs = set(os.path.normpath(path) for path in site_libs)

    site_distributions = OrderedSet()
    for path_element in sys.path:
      if any(path_element.startswith(site_lib) for site_lib in site_libs):
        TRACER.log('Inspecting path element: %s' % path_element)
        site_distributions.update(dist.location for dist in find_distributions(path_element))

    user_site_distributions = OrderedSet(dist.location for dist in find_distributions(USER_SITE))

    for path in site_distributions:
      TRACER.log('Scrubbing from site-packages: %s' % path)
    for path in user_site_distributions:
      TRACER.log('Scrubbing from user site: %s' % path)

    scrub_paths = site_distributions | user_site_distributions
    scrubbed_sys_path = list(OrderedSet(sys.path) - scrub_paths)
    scrub_from_importer_cache = filter(
      lambda key: any(key.startswith(path) for path in scrub_paths),
      sys.path_importer_cache.keys())
    scrubbed_importer_cache = dict((key, value) for (key, value) in sys.path_importer_cache.items()
      if key not in scrub_from_importer_cache)
    return scrubbed_sys_path, scrubbed_importer_cache

However, it seems that in practice this eliminates egg files from withing site-packages, but leaves the site-packages directories in the sys.path itself.

From building a pex file with PEX_VERBOSE=1:

twitter.common.python.pex: Found site-library: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
twitter.common.python.pex: Inspecting path element: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
twitter.common.python.pex: Inspecting path element: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg
twitter.common.python.pex: Scrubbing from site-packages: /usr/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
twitter.common.python.pex: Scrubbing from site-packages: /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg

And here is the output of printing sys.path from running that pex:

/Library/Python/2.7/site-packages
/Library/Python/2.7/site-packages/pip-1.2.1-py2.7.egg
/Users/paul/.pex/install/CherryPy-3.2.4-py2.7.egg
/Users/paul/.pex/install/Flask-0.9-py2.7.egg
/Users/paul/.pex/install/Jinja2-2.7-py2.7.egg
/Users/paul/.pex/install/WTForms-1.0.4-py2.7.egg
/Users/paul/.pex/install/Werkzeug-0.8.3-py2.7.egg
/Users/paul/.pex/install/mongoengine-0.8.2-py2.7.egg
/Users/paul/.pex/install/numpy-1.7.1-py2.7-macosx-10.8-x86_64.egg
/Users/paul/.pex/install/passlib-1.6.1-py2.7.egg
/Users/paul/.pex/install/pymongo-2.5.2-py2.7-macosx-10.8-intel.egg
/Users/paul/.pex/install/requests-1.2.3-py2.7.egg
/Users/paul/.pex/install/setuptools-0.9.8-py2.7.egg
/Users/paul/.pex/install/stripe-1.9.1-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex
/Users/paul/Development/urbancompass/dist/web_app.pex/.bootstrap
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Flask_Login-0.1.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Flask_Mail-0.9.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Flask_Principal-0.4.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Flask_Security-1.6.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Flask_WTF-0.8.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/MarkupSafe-0.18-py2.7-macosx-10.8-x86_64.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/Rtree-0.7.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/blinker-1.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/boto-2.13.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/dad255e5a0adef9383d42edbfadf5c4cf13a073b_src.thrift.urbancompass.badges.badges_python-dev-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/distribute-0.7.3-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/facebook_sdk-0.4.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/flask_mongoengine-0.7.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/flufl.enum-4.0-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/ipaddr-2.1.10-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/itsdangerous-0.22-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/pylibmc-1.2.3-py2.7-macosx-10.8-x86_64.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/python_statsd-1.5.7-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/pytz-2013b-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/six-1.4.1-py2.7.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/thrift-0.9.0-py2.7-macosx-10.8-x86_64.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/ujson-1.33-py2.7-macosx-10.8-x86_64.egg
/Users/paul/Development/urbancompass/dist/web_app.pex/.deps/voluptuous-0.7.2-py2.7.egg
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg
/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
/usr/local/lib/python2.7/site-packages

Which still has the following directories

/usr/local/lib/python2.7/site-packages
/Library/Python/2.7/site-packages

And definitely lets you import from them:

>>> import virtualenv
>>> print virtualenv.__file__
/usr/local/lib/python2.7/site-packages/virtualenv.pyc

Here at UrbanCompass we have just patched it to exclude these further directories, happy to share, just want to make sure that our goals are aligned.

[jsirois]

Goals are definitely aligned here - the pex should be hermetic save for the interpreter and standard libs it carries along.

[wickman]

Our internal twitter.common.python has been updated numerous times since the last sync to twitter-commons, so it's possible this issue has already been fixed internally as well. We published twitter.common.python artifacts to PyPI recently based off the internal versions. I'm curious how they compare to github's twitter-commons at this point (I'd love to just move the development of these out in the open, which seems more likely to happen when pantsbuild is split out of twitter-commons.)