easybuild-easyblocks
easybuild-easyblocks copied to clipboard
TensorFlow installation contains resolved paths in setuptools.pth
After installation of TensorFlow 1.13.1 with fosscuda/2018a and fosscuda/2019a, the paths recorded in lib/python-3.*/site-packages/setuptools.pth seem to contain resolved symlink paths.
The user-facing part of our installation is in /hpc2n/eb, which is a symlink somewhere into the /cvmfs tree.
For example:
The file /hpc2n/eb/software/MPI/GCC-CUDA/7.3.0-2.30-9.2.88/OpenMPI/3.1.1/TensorFlow/1.13.1-Python-3.6.6/lib/python3.6/site-packages/setuptools.pth contains /cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC-CUDA/7.3.0-2.30-9.2.88/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/site-packages/setuptools-40.0.0-py3.6.egg.
The file /hpc2n/eb/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/TensorFlow/1.13.1-Python-3.7.2/lib/python3.7/site-packages/setuptools.pth contains /cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/Compiler/GCCcore/8.2.0/Python/3.7.2/lib/python3.7/site-packages.
This is a significant problem as Python's site module which is responsible for adding paths from .pth files is documented to only add paths once, but as these paths differ textually to the intended /hpc2n/eb-based site-packages path, it is added redundantly as part of PYTHONPATH processing.
In the 3.7.2 example above, this breaks pip and other packages when installed as part of a virtualenv as a copy of the Python module's site-packages entry with the resolved paths ends up earlier in the sys.path than the virtualenv's own install tree.
My virtualenvs are created with virtualenv --system-site-packages to make the modules loaded visible in the virtualenv.
@zao Is this still a problem with recent TensorFlow installations?