Scope

Installing these on host OS outside of a VENV can break many other tools
If in VENV, we upgrade them to the latest just in case
Remove focal workaround now that upstream fixed it

Details

Reverts parts of ArduPilot/ardupilot#27689 because upstream https://github.com/pypa/setuptools/issues/4511 is now fixed, and this change is the root cause of https://github.com/ArduPilot/ardupilot/issues/27925

Instead of a plain revert, I think it's safer and more stable to use the OS supplied version by default. The maintainers approved this approach in https://github.com/ArduPilot/ardupilot/pull/28035

Sep 10 '24 03:09 Ryanf55

Debian buster has an issue: https://github.com/ArduPilot/ardupilot/actions/runs/10786348034/job/29913069056?pr=28056#step:5:2878

I don't have a fix in mind yet.

docker build . -t ardupilot --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) --build-arg BASE_IMAGE=debian --build-arg TAG=buster
docker run --rm -it -v "$(pwd):/ardupilot" -u "$(id -u):$(id -g)" ardupilot:latest bash

That said, buster is EOL. Perhaps if that's a blocker, we kill it from setup? https://www.debian.org/releases/buster/

Edit: Using lxml through apt solves this one too.

Sep 11 '24 04:09 Ryanf55

I had to move pygame too. We could manually install all the dependencies through apt, and then compile python3-pygame, but this seems more stable.

134.5 + for PACKAGE in $PYTHON_PKGS
134.5 + pip3 install --user -U pygame
135.0 Collecting pygame
138.5   Downloading https://files.pythonhosted.org/packages/72/49/bd2fcbadb6a55bb24284bad4f530189401c99ffc234d51ba54756a776eb2/pygame-2.6.0.tar.gz (15.8MB)
142.2     Complete output from command python setup.py egg_info:
142.2     Skipping Cython compilation
142.2     
142.2     
142.2     WARNING, No "Setup" File Exists, Running "buildconfig/config.py"
142.2     Using UNIX configuration...
142.2     
142.2     /bin/sh: 1: sdl2-config: not found
142.2     /bin/sh: 1: sdl2-config: not found
142.2     /bin/sh: 1: sdl2-config: not found
142.2     Package freetype2 was not found in the pkg-config search path.
142.2     Perhaps you should add the directory containing `freetype2.pc'
142.2     to the PKG_CONFIG_PATH environment variable
142.2     No package 'freetype2' found
142.2     Package freetype2 was not found in the pkg-config search path.
142.2     Perhaps you should add the directory containing `freetype2.pc'
142.2     to the PKG_CONFIG_PATH environment variable
142.2     No package 'freetype2' found
142.2     Package freetype2 was not found in the pkg-config search path.
142.2     Perhaps you should add the directory containing `freetype2.pc'
142.2     to the PKG_CONFIG_PATH environment variable
142.2     No package 'freetype2' found
142.2     /bin/sh: 1: freetype-config: not found
142.2     /bin/sh: 1: freetype-config: not found
142.2     /bin/sh: 1: freetype-config: not found
142.2     Traceback (most recent call last):
142.2       File "<string>", line 1, in <module>
142.2       File "/tmp/pip-install-ba7a5qs8/pygame/setup.py", line 426, in <module>
142.2         buildconfig.config.main(AUTO_CONFIG)
142.2       File "/tmp/pip-install-ba7a5qs8/pygame/buildconfig/config.py", line 234, in main
142.2         deps = CFG.main(**kwds, auto_config=auto)
142.2       File "/tmp/pip-install-ba7a5qs8/pygame/buildconfig/config_unix.py", line 245, in main
142.2         raise RuntimeError('Unable to run "sdl-config". Please make sure a development version of SDL is installed.')
142.2     RuntimeError: Unable to run "sdl-config". Please make sure a development version of SDL is installed.
142.2     
142.2     Hunting dependencies...
142.2     WARNING: "sdl2-config" failed!
142.2     WARNING: "pkg-config freetype2" failed!
142.2     WARNING: "freetype-config" failed!
142.2     
142.2     ---
142.2     For help with compilation see:
142.2         https://www.pygame.org/wiki/CompileDebian
142.2     To contribute to pygame development see:
142.2         https://www.pygame.org/contribute.html
142.2     ---
142.2     
142.2     
142.2     ----------------------------------------
142.2 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-ba7a5qs8/pygame/

Sep 11 '24 04:09 Ryanf55

So it sounds like we need to test every distro we support in there or explicitly decide to remove support.

buster can probably go; nobody should be creating a new environment on it.

Sep 11 '24 11:09 peterbarker

So it sounds like we need to test every distro we support in there or explicitly decide to remove support.

buster can probably go; nobody should be creating a new environment on it.

As part of this PR, or another? I'd really like to fix the breaking of ROS environments ASAP.

Sep 11 '24 23:09 Ryanf55

As part of this PR, or another? I'd really like to fix the breaking of ROS environments ASAP.

So it sounds like we need to test every distro we support in there or explicitly decide to remove support. buster can probably go; nobody should be creating a new environment on it.

So it sounds like we need to test every distro we support in there or explicitly decide to remove support. buster can probably go; nobody should be creating a new environment on it.

As part of this PR, or another? I'd really like to fix the breaking of ROS environments ASAP.

Anything which can be affected by this PR needs to be tested. This is why I commented on your change to affect things which don't use venv - it seemed to require a lot more testing than the alternative.

Seems like you worked out the problems with buster.

Sep 12 '24 05:09 peterbarker

As part of this PR, or another? I'd really like to fix the breaking of ROS environments ASAP.

So it sounds like we need to test every distro we support in there or explicitly decide to remove support. buster can probably go; nobody should be creating a new environment on it.

So it sounds like we need to test every distro we support in there or explicitly decide to remove support. buster can probably go; nobody should be creating a new environment on it.

As part of this PR, or another? I'd really like to fix the breaking of ROS environments ASAP.

Anything which can be affected by this PR needs to be tested. This is why I commented on your change to affect things which don't use venv - it seemed to require a lot more testing than the alternative.

Yep, this workflow of this combination of apt deps is working in an isolated Ubuntu 22 environment with ROS 2. I've tested mavproxy, sim_vehicle.py with gdb, console, map in both Ubuntu 20, Ubuntu 22, and debian buster.

Is there a better alternative you had in mind that avoids installing the latest version through pip outside a venv?

Seems like you worked out the problems with buster.

Yep!

Sep 12 '24 05:09 Ryanf55

Have you tested a Bionic install?

Yes, it's EOL. But we don't want to break things without knowing we have.

Which is why I was trying to suggest structuring this change such that it is clearly only affecting e.g. Noble, and nothing else.

That makes it easy to test. And we avoid the "ah, crap, sorry" moments when someone needs to install on something which we accidentally broke.

The older code should naturally age-out - and that's why I asked about taking buster out.

Sep 14 '24 09:09 peterbarker

I'm putting this up for dev call because

docker build . -t ardupilot --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) --build-arg BASE_IMAGE=debian --build-arg TAG=buster

More test results:

Bionic: python3-pygame is not available. As it stands, this would break bionic users.

If we want to make conditional logic to only do this on focal and jammy, I can do that.

Sep 16 '24 16:09 Ryanf55

Make a separate PR to remove bionic.

For tests, check the vagrant file for the lists of manual tests. It's extensive. We'll do a best effort to test this before merging.

Sep 17 '24 00:09 Ryanf55

Bionic is removed. Needs testing on the debian builds better.

Sep 21 '24 04:09 Ryanf55

OK, so I've now re-read this and thought about it some more.

One thing about these env install scripts - and particularly the install-prereqs-ubuntu.sh script, as it is the most complicated - is that we do not make changes that affect older distributions - not only so we don't break things, but so that people get the same sort of environment over time when they run the install scripts.

If you want to make the argument that we install apt, setuptools, wheel and whatever in the system Python environment, or via apt, or whatever, please do - but we should do that in, for example, plucky, not lunar!

I do have some painful experience with wxpython in the system environment vs in venv, so I do know there's a lot of pain to be had in this area :-)

May 11 '25 08:05 peterbarker

Previous behavior was to install apt with setuptools. See here for Plane4.4. https://github.com/ArduPilot/ardupilot/blob/3e039eb372b196dc169de25e138af6dcab28269b/Tools/environment_install/install-prereqs-ubuntu.sh#L162

I'd prefer to just do that. This needs fixed on Jammy and Noble for ROS users of LTS distros. I don't find your argument of "make changes that affect older distributions" valid - well what I am proposing is going back to the way we did it before on 4.4.

An alternative approach I proposed is to allow ardupilot on Jammy install in a venv. Then it can avoid breaking system setuptools. https://github.com/ArduPilot/ardupilot/issues/26137#issuecomment-2869637459

May 11 '25 21:05 Ryanf55

Tools: Install setuptools, packaging and wheel (and others) through apt

Scope

Details