setup.py deprecation warning

[alsora]

Hi, when building Python packages in ROS 2 Galactic the following warning is produced

--- stderr: examples_rclpy_pointcloud_publisher
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< examples_rclpy_pointcloud_publisher [1.28s]

is there a plan to change the installation logic to get rid of it? I'm not expert at all of python packaging, but would it be sufficient to replace the setup.py calls with python -m build ?

[russkel]

Looks like the Python build process should be moved over to something that follows PEP 517 (https://www.python.org/dev/peps/pep-0517/)

build is probably a good choice.

[ruffsl]

I'm testing some colcon extensions on Ubuntu Jammy 22.04, but noticed the same spammy deprecation notice:

Starting >>> colcon-ros
--- stderr: colcon-cd
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---

Guessing we should stop directly executing setup.py files and be call a dedicated python build backend instead?

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

[ksuszka]

At our project we try to follow strategy "no warnings during builds/treat all warnings as errors". Is there a way to change our python packages structure/setup.py content/whatever, to remove those warnings? Do we need to change something in our packages or is this an issue only with colcon and we need to live with those warnings till a fix in colcon is implemented?

[russkel]

You could try and install the master branch of https://github.com/colcon/colcon-python-setup-py via pip and see if that solves the issue.

[ksuszka]

AFAIK I already have the latest version (0.2.7):

root@af77d7851fa2:/ws# pip3 install colcon-python-setup-py -U
Requirement already up-to-date: colcon-python-setup-py in /usr/lib/python3/dist-packages (0.2.7)
Requirement already satisfied, skipping upgrade: colcon-core>=0.6.1 in /usr/lib/python3/dist-packages (from colcon-python-setup-py) (0.8.2)
root@af77d7851fa2:/ws# colcon build --packages-up-to as_ros_license
Starting >>> as_ros_license
--- stderr: as_ros_license                   
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< as_ros_license [2.51s]

Summary: 1 package finished [2.78s]
  1 package had stderr output: as_ros_license
root@af77d7851fa2:/ws# 

This issue doesn't occur for setuptools 45.2.0 (which are rather old but still used in the latest official ros:galactic image), however it occurs for setuptools >= 60.

[russkel]

What's on pypi is behind the master branch.

[ksuszka]

Ok, so another try:

root@23e85568dac8:/ws# pip3 install -U git+https://github.com/colcon/colcon-python-setup-py.git@master
Collecting git+https://github.com/colcon/colcon-python-setup-py.git@master
  Cloning https://github.com/colcon/colcon-python-setup-py.git (to revision master) to /tmp/pip-req-build-mt1uugi5
  Running command git clone -q https://github.com/colcon/colcon-python-setup-py.git /tmp/pip-req-build-mt1uugi5
Requirement already satisfied, skipping upgrade: colcon-core>=0.6.1 in /usr/lib/python3/dist-packages (from colcon-python-setup-py==0.2.7) (0.8.2)
Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.8/dist-packages (from colcon-python-setup-py==0.2.7) (62.1.0)
Building wheels for collected packages: colcon-python-setup-py
  Building wheel for colcon-python-setup-py (setup.py) ... done
  Created wheel for colcon-python-setup-py: filename=colcon_python_setup_py-0.2.7-py3-none-any.whl size=11692 sha256=7757e6012eb1146ff7fc3af8077e94a73aba82f21fcf22854dcf7932e2bfc832
  Stored in directory: /tmp/pip-ephem-wheel-cache-dmm0bbmb/wheels/bc/46/d5/da5ce83806c2ab78193e6ee022996337a912af8447706ea89a
Successfully built colcon-python-setup-py
Installing collected packages: colcon-python-setup-py
  Attempting uninstall: colcon-python-setup-py
    Found existing installation: colcon-python-setup-py 0.2.7
    Uninstalling colcon-python-setup-py-0.2.7:
      Successfully uninstalled colcon-python-setup-py-0.2.7
Successfully installed colcon-python-setup-py-0.2.7
root@23e85568dac8:/ws# colcon build --packages-up-to as_ros_license
Starting >>> as_ros_license
--- stderr: as_ros_license
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< as_ros_license [2.70s]

Summary: 1 package finished [3.03s]
  1 package had stderr output: as_ros_license
root@23e85568dac8:/ws#

I'm probably missing something.

[ksuszka]

I've prepared reproducible example using docker: with Dockerfile:

FROM ros:galactic

RUN apt-get update -y && apt-get install -y python3-pip
RUN pip3 install -U setuptools
RUN pip3 install -U git+https://github.com/colcon/colcon-python-setup-py.git@master
RUN mkdir -p /ws/src
WORKDIR /ws
RUN git clone https://github.com/ros2/examples.git --branch galactic /ws/src/examples
RUN colcon build --packages-select examples_rclpy_minimal_publisher

when I run command

docker build . --progress plain --no-cache

I get the same warning at the end:

#11 [8/8] RUN colcon build --packages-select examples_rclpy_minimal_publisher
#11 sha256:3365c76b7b5b832d03997e1d462f1dd7248f09be99b86c49fc69b54590821762
#11 0.780 Starting >>> examples_rclpy_minimal_publisher
#11 3.421 --- stderr: examples_rclpy_minimal_publisher
#11 3.421 /usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
#11 3.421   warnings.warn(
#11 3.421 ---
#11 3.421 Finished <<< examples_rclpy_minimal_publisher [2.64s]
#11 3.443
#11 3.443 Summary: 1 package finished [2.88s]
#11 3.443   1 package had stderr output: examples_rclpy_minimal_publisher
#11 DONE 3.5s

[russkel]

I'm probably missing something.

No probably not, it was just a suggestion. Good to know it still isn't fixed.

[jacobperron]

IIUC, we can continue to use setuptools as the build backend though we should stop invoking the setup.py CLI (which is deprecated, hence the warning).

I suppose since setuptools supports PEP 517, we could have colcon make appropriate calls to the setuptools library, acting as a build frontend. I think this would require packages built by colcon to supply most of the project metadata in a setup.cfg, instead of the setup.py (reference).

However, it appears the Python community is moving towards using pyproject.toml files for describing package metadata (PEP 621). Also, I think a pyproject.toml may be necessary to leverage modern build front-ends like build, which may make colcon's job easier (e.g. if we replaced setup.py CLI calls with build's CLI). Switching to pyproject.toml files everywhere would probably cause a lot more churn than some other solution, I don't know. It might be worth noting setuptools supports pyproject.toml (instead of setup.cfg), however it is "experimental" (reference). At the very least, it'd probably be nice to have colcon plugin to support packages using a pyproject.toml. It looks like someone has worked on one to support poetry.

Whatever the ultimate solution is, it looks like it may involve updating downstream Python packages still relying on invoking the setup.py CLI. We can keep colcon's current implementation for building Python packages as fallback behavior if the downstream package is not updated to use a static description (e.g. setup.cfg or pyproject.toml).

---

In the meantime, users can explicitly quiet the deprecation warning if it bothers them with a filter. For example, by setting the following environment variable:

PYTHONWARNINGS=ignore:::setuptools.command.install

[felixdivo]

In addition, I get /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools on ROS Humble @ Ubuntu 22.04. I can't find a place where we directly use easy_install though. It this also a problem with colcon?

[jacobperron]

I've noticed the EasyInstallDeprecationWarning deprecation warning, along with PkgResourcesDeprecationWarning, if you use colcon's --symlink-install option. If you want, you could also quiet those warnings by appending to the warnings filter, e.g.

PYTHONWARNINGS=ignore:::setuptools.command.install,ignore:::setuptools.command.easy_install,ignore:::pkg_resources

however, ultimately I think these warnings should all be resolved when colcon can build packages without a setup.py.

[felixdivo]

Okay, thanks for the confirmation! We'll silence them an wait for a patched colon. 😃

[bperseghetti]

Still seems to be an open issue, although the package still works as expected after building:

colcon build --packages-select ros2_package --symlink-install 
Starting >>> ros2_package
--- stderr: ros2_package                  
/home/ben/.local/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/user/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/user/.local/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/user/.local/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
---
Finished <<< ros2_package [2.47s]

Summary: 1 package finished [2.75s]
  1 package had stderr output: ros2_package

@dirk-thomas is anyone currently working on this or has there been any determination on the desired direction to head in resolving it (so we can help contribute to that effort)?

[jacobperron]

I believe @cottsay is working on this issue.

[FranekStark]

any news on this? :)

[cottsay]

This is a complicated change, and I don't have a lot of good news.

The new APIs that the Python folks are pushing us towards will regress the experience here in multiple ways.

1. The new APIs will force us to create Python wheels and then extract them, meaning that there will be a compression/decompression cycle affecting build performance.
2. The new APIs don't give us a way to inspect metadata like package name and dependencies, meaning there will be some packages using pyproject.toml that we won't be able to discover or build.
3. Though there are currently changes proposed to alleviate this, the entire "develop" scenario is currently unsupported meaning that symlink installs will go from "sometimes working" to never working.

I know that folks hate warning messages. I hear you. Implementing this change right now with the tools and APIs available will likely do more harm than good. Please take comfort in the fact that even the new pyproject.toml APIs which use setuptools under the hood are relying on setuptools itself to suppress it's own warning and invoke itself similar to how we invoke it here.

If you simply can't ignore the warning, please familiarize yourself with the PYTHONWARNINGS environment variable, which you can use to specifically suppress certain warnings until we can arrive at a solution that doesn't regress things so badly.

[ralwing]

Quite a similar discussion is at ament_cmake repo: https://github.com/ament/ament_cmake/issues/382

[moyutianque]

I tried to downgrad setuptools to 58.2.0 by pip3 install setuptools==58.2.0 then colcon build --symlink-install works does not show error but ros2 test still has the following error

Finished <<< examples_rclpy_pointcloud_publisher [0.96s]
--- stderr: launch_testing_examples                       

=============================== warnings summary ===============================
launch_testing_examples/check_msgs_launch_test.py: 2 warnings
launch_testing_examples/check_multiple_nodes_launch_test.py: 2 warnings
launch_testing_examples/check_node_launch_test.py: 2 warnings
launch_testing_examples/hello_world_launch_test.py: 2 warnings
launch_testing_examples/record_rosbag_launch_test.py: 2 warnings
launch_testing_examples/set_param_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/test_flake8.py::test_flake8
test/test_flake8.py::test_flake8
  Warning: SelectableGroups dict interface is deprecated. Use select.

-- Docs: https://docs.pytest.org/en/stable/warnings.html
---
Finished <<< launch_testing_examples [16.5s]

Summary: 22 packages finished [18.6s]
  10 packages had stderr output: examples_rclpy_executors examples_rclpy_guard_conditions examples_rclpy_minimal_action_client examples_rclpy_minimal_action_server examples_rclpy_minimal_client examples_rclpy_minimal_publisher examples_rclpy_minimal_service examples_rclpy_minimal_subscriber examples_rclpy_pointcloud_publisher launch_testing_examples

[Feedback02]

I tried to downgrad setuptools to 58.2.0 by pip3 install setuptools==58.2.0 then colcon build --symlink-install works does not show error but ros2 test still has the following error

Finished <<< examples_rclpy_pointcloud_publisher [0.96s]
--- stderr: launch_testing_examples                       

=============================== warnings summary ===============================
launch_testing_examples/check_msgs_launch_test.py: 2 warnings
launch_testing_examples/check_multiple_nodes_launch_test.py: 2 warnings
launch_testing_examples/check_node_launch_test.py: 2 warnings
launch_testing_examples/hello_world_launch_test.py: 2 warnings
launch_testing_examples/record_rosbag_launch_test.py: 2 warnings
launch_testing_examples/set_param_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/test_flake8.py::test_flake8
test/test_flake8.py::test_flake8
  Warning: SelectableGroups dict interface is deprecated. Use select.

-- Docs: https://docs.pytest.org/en/stable/warnings.html
---
Finished <<< launch_testing_examples [16.5s]

Summary: 22 packages finished [18.6s]
  10 packages had stderr output: examples_rclpy_executors examples_rclpy_guard_conditions examples_rclpy_minimal_action_client examples_rclpy_minimal_action_server examples_rclpy_minimal_client examples_rclpy_minimal_publisher examples_rclpy_minimal_service examples_rclpy_minimal_subscriber examples_rclpy_pointcloud_publisher launch_testing_examples

yeah i have the same issue with Humble with ubuntu 22.04.1 even after trying what you did

[ijnek]

I tried to downgrad setuptools to 58.2.0 by pip3 install setuptools==58.2.0 then colcon build --symlink-install works does not show error but ros2 test still has the following error

Finished <<< examples_rclpy_pointcloud_publisher [0.96s]
--- stderr: launch_testing_examples                       

=============================== warnings summary ===============================
launch_testing_examples/check_msgs_launch_test.py: 2 warnings
launch_testing_examples/check_multiple_nodes_launch_test.py: 2 warnings
launch_testing_examples/check_node_launch_test.py: 2 warnings
launch_testing_examples/hello_world_launch_test.py: 2 warnings
launch_testing_examples/record_rosbag_launch_test.py: 2 warnings
launch_testing_examples/set_param_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/test_flake8.py::test_flake8
test/test_flake8.py::test_flake8
  Warning: SelectableGroups dict interface is deprecated. Use select.

-- Docs: https://docs.pytest.org/en/stable/warnings.html
---
Finished <<< launch_testing_examples [16.5s]

Summary: 22 packages finished [18.6s]
  10 packages had stderr output: examples_rclpy_executors examples_rclpy_guard_conditions examples_rclpy_minimal_action_client examples_rclpy_minimal_action_server examples_rclpy_minimal_client examples_rclpy_minimal_publisher examples_rclpy_minimal_service examples_rclpy_minimal_subscriber examples_rclpy_pointcloud_publisher launch_testing_examples

yeah i have the same issue with Humble with ubuntu 22.04.1 even after trying what you did

This is an unrelated flake8 issue that is tracked in https://bugs.launchpad.net/ubuntu/+source/python-flake8/+bug/1968072, so it has nothing to do with this ticket.

[cottsay]

We'll be discussing this topic at the Infrastructure Community Meeting on May 17: https://discourse.ros.org/t/infrastructure-community-meeting-2023-05-17/31268

[cottsay]

I've issued a Call For Testing on a prototype package based on the PEPs discussed here: https://discourse.ros.org/t/call-for-testing-standards-based-python-packaging-with-colcon/32008

If you're interested in seeing these deprecation warnings disappear, please take a moment to provide feedback.

[Flova]

Whats the current state of this issue?

[cottsay]

The prototype is available and working, but there are problems with symlink installs and general performance. PEP 660 with setuptools is currently ignoring data_files, which is the mechanism that ROS packages use to install things like package manifests and launch files. That alone is a blocker.

If you're interested in this landing sooner, please consider trying the prototype package and providing feedback.

[TheMarksniper]

The prototype is available and working, but there are problems with symlink installs and general performance. PEP 660 with setuptools is currently ignoring data_files, which is the mechanism that ROS packages use to install things like package manifests and launch files. That alone is a blocker.

If you're interested in this landing sooner, please consider trying the prototype package and providing feedback.

Is there any way to download this prototype via pip?

[cottsay]

Is there any way to download this prototype via pip?

You can target the git branches directly using pip, but I don't think it's a good idea at all. Pip has a tendency to tangle up your system and it might make it difficult to revert back to the non-prototype packages.

[rafa-martin]

As in PR #626, it would be beneficial to suppress this error by default as well:

/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---

https://github.com/colcon/colcon-core/blob/98e30ff0cf84d40dedca6726409304105c1390e5/colcon_core/task/python/build.py#L28-L32 The updated implementation could look like this:

_PYTHON_CMD = [
    sys.executable,
    '-W',
    'ignore:setup.py install is deprecated',
    '-W',
    'ignore:easy_install command is deprecated',
]

Would it make sense to suppress this error too?"