easybuild-easyblocks icon indicating copy to clipboard operation
easybuild-easyblocks copied to clipboard

TensorFlow installation contains resolved paths in setuptools.pth

Open zao opened this issue 6 years ago • 1 comments

After installation of TensorFlow 1.13.1 with fosscuda/2018a and fosscuda/2019a, the paths recorded in lib/python-3.*/site-packages/setuptools.pth seem to contain resolved symlink paths.

The user-facing part of our installation is in /hpc2n/eb, which is a symlink somewhere into the /cvmfs tree.

For example: The file /hpc2n/eb/software/MPI/GCC-CUDA/7.3.0-2.30-9.2.88/OpenMPI/3.1.1/TensorFlow/1.13.1-Python-3.6.6/lib/python3.6/site-packages/setuptools.pth contains /cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC-CUDA/7.3.0-2.30-9.2.88/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/site-packages/setuptools-40.0.0-py3.6.egg.

The file /hpc2n/eb/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/TensorFlow/1.13.1-Python-3.7.2/lib/python3.7/site-packages/setuptools.pth contains /cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/Compiler/GCCcore/8.2.0/Python/3.7.2/lib/python3.7/site-packages.

This is a significant problem as Python's site module which is responsible for adding paths from .pth files is documented to only add paths once, but as these paths differ textually to the intended /hpc2n/eb-based site-packages path, it is added redundantly as part of PYTHONPATH processing.

In the 3.7.2 example above, this breaks pip and other packages when installed as part of a virtualenv as a copy of the Python module's site-packages entry with the resolved paths ends up earlier in the sys.path than the virtualenv's own install tree.

My virtualenvs are created with virtualenv --system-site-packages to make the modules loaded visible in the virtualenv.

zao avatar Oct 25 '19 13:10 zao

@zao Is this still a problem with recent TensorFlow installations?

boegel avatar Sep 30 '21 09:09 boegel