Setup script produces a long PYTHONPATH with --symlink-install + --merge-install

[rotu]

When running colcon build --merge-install environment variable paths generally get shorter as packages are merged into a common space. Unfortunately, when combining this with --symlink-install, PYTHONPATH gets extended with an entry for each package.

It should be unnecessary to add this to unnecessary since there are already egg links inside install/lib/python3.6/site-packages. BUT most of these packages seem absent from easy-install.pth.

$ source install/setup.sh
$ pip3 list | grep ament
ament-copyright            0.7.4               /opt/ros/master/build/ament_copyright 
ament-flake8               0.7.4               /opt/ros/master/build/ament_flake8    
ament-lint                 0.7.4               /opt/ros/master/build/ament_lint      
ament-lint-cmake           0.7.4               /opt/ros/master/build/ament_lint_cmake
ament-pep257               0.7.4               /opt/ros/master/build/ament_pep257    

$ PYTHONPATH=/opt/ros/master/install/lib/python3.6/site-packages
$ pip3 list | grep ament
ament-lint-cmake           0.7.4               /opt/ros/master/build/ament_lint_cmake
$ cat install/lib/python3.6/site-packages/easy-install.pth 
/opt/ros/master/build/ament_lint_cmake

Below is the entire environment produced by calling colcon build --packages-up-to ament_lint_cmake (with and without symlink/merge install), and then env -i - PATH=/usr/bin /bin/bash -c "source install/setup.bash; env"

# default (no symlink/no merge)

AMENT_PREFIX_PATH=/opt/ros/master/install/ament_pep257:/opt/ros/master/install/ament_lint_cmake:/opt/ros/master/install/ament_flake8:/opt/ros/master/install/ament_copyright:/opt/ros/master/install/ament_lint
PWD=/opt/ros/master
COLCON_PREFIX_PATH=/opt/ros/master/install
SHLVL=1
PYTHONPATH=/opt/ros/master/install/ament_pep257/lib/python3.6/site-packages:/opt/ros/master/install/ament_lint_cmake/lib/python3.6/site-packages:/opt/ros/master/install/ament_flake8/lib/python3.6/site-packages:/opt/ros/master/install/ament_copyright/lib/python3.6/site-packages:/opt/ros/master/install/ament_lint/lib/python3.6/site-packages
PATH=/opt/ros/master/install/ament_pep257/bin:/opt/ros/master/install/ament_lint_cmake/bin:/opt/ros/master/install/ament_flake8/bin:/opt/ros/master/install/ament_copyright/bin:/usr/bin
_=/usr/bin/env

# symlink

AMENT_PREFIX_PATH=/opt/ros/master/install/ament_pep257:/opt/ros/master/install/ament_lint_cmake:/opt/ros/master/install/ament_flake8:/opt/ros/master/install/ament_copyright:/opt/ros/master/install/ament_lint
PWD=/opt/ros/master
COLCON_PREFIX_PATH=/opt/ros/master/install
SHLVL=1
PYTHONPATH=/opt/ros/master/build/ament_pep257:/opt/ros/master/install/ament_pep257/lib/python3.6/site-packages:/opt/ros/master/build/ament_lint_cmake:/opt/ros/master/install/ament_lint_cmake/lib/python3.6/site-packages:/opt/ros/master/build/ament_flake8:/opt/ros/master/install/ament_flake8/lib/python3.6/site-packages:/opt/ros/master/build/ament_copyright:/opt/ros/master/install/ament_copyright/lib/python3.6/site-packages:/opt/ros/master/build/ament_lint:/opt/ros/master/install/ament_lint/lib/python3.6/site-packages
PATH=/opt/ros/master/install/ament_pep257/bin:/opt/ros/master/install/ament_lint_cmake/bin:/opt/ros/master/install/ament_flake8/bin:/opt/ros/master/install/ament_copyright/bin:/usr/bin
_=/usr/bin/env

# merge

AMENT_PREFIX_PATH=/opt/ros/master/install
PWD=/opt/ros/master
COLCON_PREFIX_PATH=/opt/ros/master/install
SHLVL=1
PYTHONPATH=/opt/ros/master/install/lib/python3.6/site-packages
PATH=/opt/ros/master/install/bin:/usr/bin
_=/usr/bin/env

# symlink + merge

AMENT_PREFIX_PATH=/opt/ros/master/install
PWD=/opt/ros/master
COLCON_PREFIX_PATH=/opt/ros/master/install
SHLVL=1
PYTHONPATH=/opt/ros/master/build/ament_pep257:/opt/ros/master/build/ament_lint_cmake:/opt/ros/master/build/ament_flake8:/opt/ros/master/build/ament_copyright:/opt/ros/master/build/ament_lint:/opt/ros/master/install/lib/python3.6/site-packages
PATH=/opt/ros/master/install/bin:/usr/bin
_=/usr/bin/env

[dirk-thomas]

I don't think there is anything which can be done about it. Please see #163 why the package specific build directory must be on the PYTHONPATH.

[rotu]

From what you pointed me to, it sounds like using pth files puts things in the wrong order with respect to overlayed workspaces. I'm not sure I understand from that conversation in #163 or #152 what actually happens with when both an underlay and overlay have their own easy-install.pth and site.py files.

It does seem weird that we're clobbering the easy-install.pth file on each build - this file should probably contain either all packages in the workspace or none at all.

And maybe the packages should be symlinked into site-packages. Egg links exist because some platforms don't support native symlinks. Presumably we can use real symlinks and just point PYTHONPATH to site-packages.

[dirk-thomas]

colcon is not doing anything custom here. It simply invokes setup.py develop. Since I don't think we want to implement a custom "develop" logic in colcon the resulting layout / files are kind of fixed and we have to deal with it.

[rotu]

That's only partly true. It seems the --editable flag to setup.py develop allows us to deactivate the code for easy-install.pth (#219), I presume because easy-install is only suitable for packages installed at either the site-level, user-level, or venv-level, and editable packages exist outside those places.

Also, take a look at https://github.com/RoverRobotics-forks/colcon-core/commit/c58a329b06da94f53055c4ba6f01d82171285fda . It's not ready for primetime yet, but does it seem like a plausible approach?

[dirk-thomas]

take a look at RoverRobotics-forks@c58a329 . It's not ready for primetime yet, but does it seem like a plausible approach?

How is this supposed to work on Windows?

[rotu]

How is this supposed to work on Windows?

What do you mean? If you’re already passing the —symlink-install flag, you’re letting Colcon know that the system supports symlinks, no?

[dirk-thomas]

Python setuptools develop does work on Windows since it doesn't use symlinks. The logic in colcon should gracefully fall back if the system doesn't support symlinks.

[rotu]

This makes no more assumptions about the viability of symlinks than the surrounding code. Currently, we symlink the sources to the build directory immediately prior to calling setup.py develop.

https://github.com/RoverRobotics-forks/colcon-core/blob/c58a329b06da94f53055c4ba6f01d82171285fda/colcon_core/task/python/build.py#L86

[dirk-thomas]

I see your point now (I was under the false impression that logic was conditional). That approach might be feasible then. Please go ahead and create a pull request once its ready and I am happy to try and review it.