tree-wide: port debpython to termux

[robertkirkman]

• Fixes https://github.com/termux/termux-packages/issues/25921

• Fixes https://github.com/termux/termux-packages/issues/21658

What is debpython?

• debpython is the commands py3compile and py3clean from Debian. I am calling them that because a large chunk of their source code is found inside a folder inside Debian's source code named "debpython"

  • https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5

• rather than packaging .pyc files into .deb files, py3compile and py3clean can be called from postinst and prerm scripts to generate all .pyc for the .py files in the package immediately after the package is installed, and remove all .pyc files immediately before uninstalling the package, respectively

• fixes the error trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc' when packages were built on-device, but at the same time, also:

• prevents the warnings dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed, as long as no packages were installed using pip.

• Not implemented for pypy, pypy3, or python2 packages, only python3 packages currently.

How to add a new Python package after this?

Everything is the same, except, now, this block is no longer necessary in build.sh.

termux_step_create_debscripts() {
	cat <<- EOF > ./postinst
	#!$TERMUX_PREFIX/bin/sh
	echo "Installing dependencies through pip..."
	pip3 install ${TERMUX_PKG_PYTHON_TARGET_DEPS//, / }
	EOF
}

• Instead, scripts/build/termux_step_create_python_debscripts.sh can now detect the presence of pip package lists in $TERMUX_PKG_PYTHON_TARGET_DEPS, $TERMUX_SUBPKG_PYTHON_TARGET_DEPS, and the METADATA file of the Python package if it exists, and automatically insert them as a block into the postinst script for all relevant packages.
• $TERMUX_PKG_PYTHON_TARGET_DEPS is used for pip dependencies that are both on-device build-time and on-device run-time dependencies, and $TERMUX_PKG_PYTHON_RUNTIME_DEPS is used for runtime-only pip dependencies. $TERMUX_PKG_PYTHON_RUNTIME_DEPS overrides $TERMUX_PKG_PYTHON_TARGET_DEPS for runtime dependencies,
  • i.e. if TERMUX_PKG_PYTHON_RUNTIME_DEPS is not specified, but TERMUX_PKG_PYTHON_TARGET_DEPS is, then TERMUX_PKG_PYTHON_TARGET_DEPS will be used as both on-device build and on-device runtime dependencies,
  • but if TERMUX_PKG_PYTHON_RUNTIME_DEPS is specified, then TERMUX_PKG_PYTHON_TARGET_DEPS, if specified, is used only for on-device build-time dependencies.
• python-pip is also automatically inserted into the runtime dependencies of every package that needs it, appropriately.

[licy183]

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt. It is quite annoying to see some warnings like xxx directory is not empty.

[robertkirkman]

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt

Should the content in these folders remain managed by apt in any case?

In Debian, I believe these folders are not managed by apt.

https://packages.debian.org/sid/amd64/libpython3.12-minimal/filelist

In Arch Linux, I believe these folders are managed by pacman.

https://archlinux.org/packages/core/x86_64/python/ (file list)

Fedora also ships them

https://packages.fedoraproject.org/pkgs/python3.12/python3-libs/fedora-40.html

it seems to me like it is optional whether to include them in the package, because I think Python can automatically generate them, but this problem is very difficult because, I think Arch Linux and Fedora might ship them for performance.

At least this does not break Python, I think, because a python package built from this change and installed in Termux after rm -rf $PREFIX/lib/python3.12 still works and runs.

It is quite annoying to see some warnings like xxx directory is not empty.

I do not think there is any way to get rid of this warning, because on Termux python-pip generates and stores the __pycache__ folders into the site-packages folder in the same $PREFIX/lib/python3.12/ folder where the $PREFIX/lib/python3.12/zoneinfo/_zoneinfo.py files are, etc.

Example: warning always reproducible with current packages

pkg install python-pip
pip install setuptools
pkg remove python

Removing python-pip (25.0.1) ...
Removing python-ensurepip-wheels (3.12.9) ...
Removing python (3.12.9) ...
dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed
Processing triggers for man (1.14.6-1) ...

That happens without this change.

Can you think of any way to:

• prevent the __pycache__ folders from being incorrectly included in non-Python packages during on-device build-package.sh, to fix trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc', which is also in package python 3.12.9
• but still make sure __pycache__ folders are included in the package only when they actually belong to the package currently being built, so they can be shipped for a Python startup performance boost?

I do not actually notice the performance difference at the startup of Python, when I try to test for it, but I believe someone else might.

[licy183]

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

[robertkirkman]

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

ok I did it, is this what you meant?

[robertkirkman]

@Maxython Ok, to continue the discussion here,

now I have:

• unified almost all python debscript creation into a single function, removing the duplicated code involving $TERMUX_PKG_PYTHON_TARGET_DEPS from many packages and also making it not necessary to explicitly call termux_step_create_python_debscripts() from any build.sh anymore
• tried to make this debpython compatible with pacman by making it try to use the pacman -Qql command if the dpkg command is not found

however, I am not sure whether the pacman -Qql command is completely compatible to the point that it will work in all cases without problems. pacman -Qql does print equivalent data in very similar formatting to dpkg -L, but it is not 100% identical formatting to dpkg -L, so I am not sure whether it will work perfectly without a lot of testing.

[robertkirkman]

That's ok, something I have noticed is that, licy183 told me somewhere else that TERMUX_PKG_REVISION should not be incremented for PRs that are excluded from CI. The reason I excluded the PR from CI is because I previously learned here that if too many packages are built in a single PR, sometimes the behavior will diverge from the behavior of the same large build command run in the docker builder locally (even when packages are not polluting the builds of other packages)

However, the reason why many packages are marked with revision-bumps in this is because, that is marking all packages that a test I did indicates will have changed control.tar.xz contents if they are rebuilt after this PR is merged.

Because of that, I think probably what I need to do is split this into several PRs one after another that do not have CI disabled.

[Maxython]

Update: The provided list of packages that need to be rebuilt is incomplete, i.e. it does not include all the packages that need to be rebuilt with debpython.

[TomJo2000]

The following packages need to be recompiled using debpython:

I would suggest either opening a staging PR for this purpose, or adding the commits for rebuilding those packages to this PR.

[Maxython]

I would suggest either opening a staging PR for this purpose, or adding the commits for rebuilding those packages to this PR.

It is planned to rebuild these packages in each PRs. If rebuild all the packages in one PR (for example on this one), there are risks that can break the compilation.

[TomJo2000]

It is planned to rebuild these packages in each PRs. If rebuild all the packages in one PR (for example on this one), there are risks that can break the compilation.

I'm not sure I understand that concern.

[Maxython]

I'm not sure I understand that concern.

Well, there are quite a lot of packages that need to be rebuilt. And one of the risks I see is that the packages may not have time to be built, i.e. their compilation may take more than six hours.

[TomJo2000]

Those 18 packages should get nowhere near 6 hours. The largest one there is pyqt5 which took ~20 minutes the last time it was built.

[robertkirkman]

Unfortunately, I'm unable to reproduce the matplotlib build error here locally. For me locally, all packages other than python-torchvision cross-compile successfully, but in CI, every time it fails to build matplotlib. This means there is a difference between my computer and CI for that package, I will try to find out what it is.

[robertkirkman]

Yes so, Tomjo2000, what we're interested in is whether all these packages will install and work successfully on-device after building here, and without showing the message:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

[robertkirkman]

Compatibility with CMake < 3.5 has been removed from CMake.

After

• #25014

CMake in all of Termux is now CMake 4, so packages that require CMake 3 cannot build anymore until they receive updates.

[robertkirkman]

I have been testing this PR more and adding more necessary changes in certain places, I have added my newest changes as an additional temporary commit here so that Maxython can easily distinguish this version of the PR from the previous version of the PR.

After the latest changes, there are no fatal errors during cross-compilation of all packages, installation of all packages on-device, and basic launch tests of some of the programs,

but there are still some non-fatal errors during installation, and some fatal errors during on-device building, and not all of those fatal errors during on-device building are exactly the same with and without this PR - meaning that unfortunately, this PR does add some additional errors to on-device building, but only to packages that couldn't be built on-device previously anyway.

Since on-device building is lower-priority than cross-compilation, it seems unclear how big of a problem it is that this adds additional errors to the on-device builds of packages that already wouldn't build on-device - (all packages modified here that could already build successfully on-device, still seem to build successfully on-device in this PR),

but in my opinion, probably it will be best to at least try to figure out some things about the unique, new errors being added for on-device building to three packages (python-onnxruntime, python-torch, python-torchvision) and try to do something to prevent the errors.

Fatal errors during standard cross-compilation and installation through package manager

• No known fatal errors currently remaining in this category at the most recent commit

Errors during on-device building

Packages that have the same error during on-device building both with and without this PR

• python
• pyqt5
• matplotlib

Packages that have an error during on-device building both with and without this PR, but the error is different between them

[!TIP] these errors are the ones that I knew about that were the reasons I didn't originally touch packages like python-onnxruntime in this PR, but there are a bit fewer than I originally hypothesized. Only three packages seem to be truly affected by the frightening problem that discouraged me from changing these packages for risk of breaking them any further than they already are (I had expected more).

The errors are related to the detail that on-device building of packages installs $TERMUX_PKG_PYTHON_TARGET_DEPS before the building and installation of the package itself. I am not yet completely sure what the best changes to this PR to prevent these errors should be (assuming that changes should attempt to preserve the idea of Maxython of putting all pip dependencies in the $TERMUX_PKG_PYTHON_TARGET_DEPS variable of these affected packages)

• python-onnxruntime
• python-torch
• python-torchvision

Other non-fatal errors

As licy183 explained, there is a large amount of messages like this:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nala 0.16.0 requires anyio<5.0.0,>=4.9.0, which is not installed.
nala 0.16.0 requires httpx==0.28.1, which is not installed.
nala 0.16.0 requires pexpect<5.0.0,>=4.9.0, which is not installed.
nala 0.16.0 requires rich<14.0.0,>=13.9.4, which is not installed.
nala 0.16.0 requires tomli<3.0.0,>=2.2.1, which is not installed.
nala 0.16.0 requires typer<0.16.0,>=0.15.2, which is not installed.
manim 0.19.0 requires av<14.0.0,>=9.0.0, which is not installed.
manim 0.19.0 requires beautifulsoup4>=4.12, which is not installed.
manim 0.19.0 requires click>=8.0, which is not installed.
manim 0.19.0 requires cloup>=2.0.0, which is not installed.
manim 0.19.0 requires decorator>=4.3.2, which is not installed.
manim 0.19.0 requires isosurfaces>=0.1.0, which is not installed.
manim 0.19.0 requires manimpango<1.0.0,>=0.5.0, which is not installed.
manim 0.19.0 requires mapbox-earcut>=1.0.0, which is not installed.
manim 0.19.0 requires moderngl<6.0.0,>=5.0.0, which is not installed.
manim 0.19.0 requires moderngl-window>=2.0.0, which is not installed.
manim 0.19.0 requires pydub>=0.20.0, which is not installed.
manim 0.19.0 requires rich>=12.0.0, which is not installed.
manim 0.19.0 requires screeninfo>=0.7, which is not installed.
manim 0.19.0 requires srt>=3.0.0, which is not installed.
manim 0.19.0 requires svgelements>=1.8.0, which is not installed.
manim 0.19.0 requires tqdm>=4.0.0, which is not installed.
manim 0.19.0 requires watchdog>=2.0.0, which is not installed.
tldp 0.7.5 requires nose, which is not installed.

Despite that, installation does not fail and the programs seem to run successfully, however, whether they have subtle errors with certain actions or not as a result of the pip errors, is unknown. The above messages do not appear when installing the current version of nala using apt install nala, but would begin appearing after this PR, I think.

[robertkirkman]

After some more testing and consideration, I have decided to add some more changes into another temporary commit here, Maxython, you can read my most recent commit in order to review my newest changes and decide whether you agree with them.

[Maxython]

LGTM

[robertkirkman]

I had been testing this and encountering this error during installation of nala

Errno 2] No such file or directory: '/data/data/com.termux/files/usr/lib/python3.12/site-packages/nala/cache.py'
dpkg: error processing package nala (--configure):

since this was technically an error producing a failing result, I created a way to prevent this error and have applied it to the recent commit, and you can let me know if you agree with the solution I created.

This will make py3compile skip compiling .py files if dpkg reports they are in the package, but they end up not being present when py3compile runs because of pip install deleting those files for some reason if the PyPi equivalent name of the software being packageed is in TERMUX_PKG_PYTHON_RUNTIME_DEPS.

After this, I think this PR content is ready, and the commits can be squashed into a single commit. What do you think @Maxython ?

I had some other ideas for additional features or changes that change even more than this, but they are too big for this PR and if I still want them I will make other PRs for them later.

[Maxython]

pip install deleting those files for some reason if the PyPi equivalent name of the software being packageed is in TERMUX_PKG_PYTHON_RUNTIME_DEPS.

It seems that this happens because during the installation of the module, pip updates it to the latest version (i.e. this happens because pip has the --upgrade flag set). And due to this update, the contents of the module's *.py files may change (i.e. they may be deleted or new ones may be created).

After this, I think this PR content is ready, and the commits can be squashed into a single commit. What do you think @Maxython ?

I think we should first reconsider the method for solving this problem you found. Then this PR can be merged with two commits (in one commit the implementation of debpython, in the other recompilation of some packages).

[robertkirkman]

I think we should first reconsider the method for solving this problem you found. Then this PR can be merged with two commits (in one commit the implementation of debpython, in the other recompilation of some packages).

Ok sure, sounds great. If you have a specific idea for how to fix the problem of Errno 2] No such file or directory: '/data/data/com.termux/files/usr/lib/python3.12/site-packages/nala/cache.py' in a better way, then feel free to force push this branch with any change you want to show, and we can see how that way looks

[Maxython]

If you have a specific idea for how to fix the problem of Errno 2] No such file or directory: '/data/data/com.termux/files/usr/lib/python3.12/site-packages/nala/cache.py' in a better way, then feel free to force push this branch with any change you want to show, and we can see how that way looks

The idea for solving the problem: if in the list of modules to install during the installation of packages there is a module whose name matches the name of the package, then the --upgrade flag will be disabled.

[robertkirkman]

Ok, I have tested installing the current version on two termux-apt devices with two different CPU architectures, and one clean installed and one not clean installed, and I can confirm there are no more fatal errors anywhere in this PR.

However, the one remaining possible problem I see is, I still frequently see these errors:

licy183 mentioned these errors first, but unfortunately, I don't know how to fix these.

They are non-fatal errors, because after they show, "Successfully installed" prints anyway.

Also, they do not necessarily produce runtime errors in the python programs, because for example, after this I tested commands like manim --show-splash init project, and they appear to work without errors.

On the other hand, I am not sure whether these errors indicate that subtle, hard-to-find crashes in some programs might appear.

Do you think it is acceptable to allow these errors, or do you know of any way to prevent them?

[robertkirkman]

I checked and noticed that the dependencies logged as "not installed", do actually appear as installed at the end after all debscripts finish running.

for example, mapbox-earcut 1.0.2 is installed after the apt install command finishes, but I didn't manually install it.

This suggests the error might be spurious, or also, it could be a coincidence, and maybe mapbox-earcut was installed by a different debscript somehow, from the one that printed that message. I'm not sure which is the case.

[licy183]

Do you think it is acceptable to allow these errors, or do you know of any way to prevent them?

As I commented before, if you install some package manually (pip install manim), pip will automatically change these dependencies to correct version.

[robertkirkman]

Thank you for explaining, I am sorry for being dense about this detail. I think I am beginning to get an idea of what should be done. I will think about this and make a solution.

[robertkirkman]

After the most recent changes, when I test fully recompiling and reinstalling every single package here at once into a clean Termux installation, the amount of visible instances of ERROR: pip's dependency resolver does not currently take into account all the packages that are installed has not been 100% completely eliminated, but they have been greatly reduced, and all of the instances remaining seem to occur because apt installs all the python package files into $PREFIX before ever running any debscripts, then runs the postinst scripts one by one for each package, so these messages sometimes print temporarily while not all packages have had their postinst scripts run yet - but I believe that all PyPi dependencies are now installed correctly by the time all postinst scripts have finished running.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
borgbackup 1.4.1 requires msgpack<=1.1.0,>=1.0.3, which is not installed.
matplotlib 3.10.3 requires cycler>=0.10, which is not installed.
matplotlib 3.10.3 requires fonttools>=4.22.0, which is not installed.
matplotlib 3.10.3 requires kiwisolver>=1.3.1, which is not installed.
matplotlib 3.10.3 requires pyparsing>=2.3.1, which is not installed.
matplotlib 3.10.3 requires python-dateutil>=2.7, which is not installed.
torch 2.6.0 requires filelock, which is not installed.
torch 2.6.0 requires fsspec, which is not installed.
torch 2.6.0 requires jinja2, which is not installed.
torch 2.6.0 requires networkx, which is not installed.
torch 2.6.0 requires setuptools; python_version >= "3.12", which is not installed.
torch 2.6.0 requires typing-extensions>=4.10.0, which is not installed.
torch 2.6.0 requires sympy==1.13.1; python_version >= "3.9", but you have sympy 1.14.0 which is incompatible.
Successfully installed coloredlogs-15.0.1 flatbuffers-25.2.10 humanfriendly-10.0 mpmath-1.3.0 packaging-25.0 protobuf-6.31.1 sympy-1.14.0
Setting up ffmpeg (7.1.1-4) ...
Setting up python-torch (2.6.0-1) ...
Installing dependencies for python-torch through pip...
Requirement already satisfied: torch in /data/data/com.termux/files/usr/lib/python3.12/site-packages (2.6.0)
Collecting filelock (from torch)
  Downloading filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.10.0 (from torch)
  Downloading typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB)
Collecting setuptools (from torch)
  Downloading setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
Collecting sympy==1.13.1 (from torch)
  Downloading sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch)
  Downloading networkx-3.5-py3-none-any.whl.metadata (6.3 kB)
Collecting jinja2 (from torch)
  Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec (from torch)
  Downloading fsspec-2025.7.0-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /data/data/com.termux/files/usr/lib/python3.12/site-packages (from sympy==1.13.1->torch) (1.3.0)

In particular, I noticed by looking closely that I think all of the dependencies are now being resolved as well as possible for every single package, one by one as the postinst scripts for each package explicitly install the pip-facing package name without --upgrade:

torch 2.6.0 requires filelock, which is not installed.

...

Setting up python-torch (2.6.0-1) ...
Installing dependencies for python-torch through pip...
Requirement already satisfied: torch in /data/data/com.termux/files/usr/lib/python3.12/site-packages (2.6.0)
Collecting filelock (from torch)
  Downloading filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)

@Maxython The amount of changes I have made to do this is quite large, and I am not sure whether you agree with all of the changes. Please let me know if you think everything I have done here is a good idea.

however, after you have finished reviewing my new changes, and any follow-up changes have finished, I believe this PR will be ready.

[robertkirkman]

The calcurse.org website is currently down, so I cannot build this PR at the moment, but it should build after that resolves eventually. EDIT: now resolved

[robertkirkman]

Basically, I need it and it's really inconvenient for me not to have all the necessary changes in one branch, so I've now added all additional changes that are required to fix

• https://github.com/termux/termux-packages/issues/21658

This means that after 10 months in development, during which many other fixes have been merged in other PRs that have whittled down more and more of the errors here and there, this PR is now a 100% solution to that issue (./build-package.sh -I -f mesa on-device), instead of only part of the solution.

[!TIP] This PR that was recently merged globally fixed a large number of on-device build errors that I had been aware of for years,

• https://github.com/termux/termux-packages/pull/25627

and it makes up a portion of the fixes for that issue, but I was unfortunately a bit too unassertive to believe that I could convincingly argue for that change to be made for a long time (which ended up being until NDK r28c broke the existing code further, giving me the opportunity to prove that my approach is better in a high-priority situation)

• https://github.com/termux/termux-packages/issues/25622

[robertkirkman]

@hansm629 I believe this PR should now form an implementation of a 100% solution to your October 4 2024 issue,

if you would like to, you can try testing it to just confirm that on your end, by using these commands, which should ideally be 100% successful without any other manual intervention, with no errors.

git clone -b exclude-pycache-during-extract-into-massagedir https://github.com/robertkirkman/termux-packages.git
cd termux-packages/
scripts/setup-termux.sh
./build-package.sh -I -f mesa
cd output/
apt reinstall ./*.deb