python: update to 3.12

[landfillbaby]

will this even work since distutils is deleted?

[landfillbaby]

idk someone else can make it work

[2096779623]

https://github.com/python/cpython/blob/6678ef41d2a188a92c7ac5249d716e942b5114a6/configure.ac#L153C50-L153C50

[licy183]

We should build a host python for python3.12. Ubuntu 22.04 hasn't packaged it yet. Some other repositories such as ppa/deadsnakes may be an alternative in the future, but it hasn't packaged python3.12 yet.

IMO we had better hold it to 3.11 until some other python libraries support it. Bumping to python3.12 will at least break the build process of numpy because python3.12 remove support for distutils but meson-python (which numpy uses for python3.12) hasn't supported cross-compile.

[romanovj]

I think that termux should stick to 3.11and 3.12 could be provided by tur-repo

[TomJo2000]

I think that termux should stick to 3.11and 3.12 could be provided by tur-repo

The opposite will be the case. Termux is a rolling release and the main repo hosts the latest versions of packages. Older versions are only provided in special cases such as LTS releases, or on the TUR.

[showierdata9978]

atleast allow people to install it as python3.12

[Randrianasulu]

installing CI artefacts resulted in pip setup error:

~ $ dpkg -i debs/python*.deb                                                                       
(Reading database ... 125977 files and directories currently installed.)                           
Preparing to unpack .../python-ensurepip-wheels_3.12.0_all.deb ...                               
  Unpacking python-ensurepip-wheels (3.12.0) over (3.12.0) ...                                       Preparing to unpack .../python-numpy-static_1.26.1-1_aarch64.deb ...                              
 Unpacking python-numpy-static (1.26.1-1) over (1.26.1-1) ...                                       Preparing to unpack .../python-numpy_1.26.1-1_aarch64.deb ...                                      Unpacking python-numpy (1.26.1-1) over (1.26.1-1) ...                                              Preparing to unpack .../python-pillow_10.1.0-1_aarch64.deb ...                                     Unpacking python-pillow (10.1.0-1) over (10.1.0-1) ...                                             Preparing to unpack debs/python-pip_23.3-1_all.deb ...                                             Unpacking python-pip (23.3-1) over (23.3-1) ...                                                    Preparing to unpack .../python-static_3.12.0_aarch64.deb ...                                       Unpacking python-static (3.12.0) over (3.12.0) ...                                                 Preparing to unpack .../python-tkinter_3.12.0_aarch64.deb ...                                      Unpacking python-tkinter (3.12.0) over (3.12.0) ...                                                Preparing to unpack debs/python_3.12.0_aarch64.deb ...                                             Unpacking python (3.12.0) over (3.12.0) ...                                                        Setting up python (3.12.0) ...                                                                     Setting up python-ensurepip-wheels (3.12.0) ...                                                    Setting up python-numpy (1.26.1-1) ...                                                             
Setting up python-pillow (10.1.0-1) ...                                                            
Setting up python-pip (23.3-1) ...                                                                 
pip setup...                                                                                       
Traceback (most recent call last):                                                                   File "/data/data/com.termux/files/usr/bin/pip", line 5, in <module>                                  
from pip._internal.cli.main import main                                                          File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 10, in <module>                                                                                     from pip._internal.cli.autocompletion import autocomplete                                        File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>                                                                          
 from pip._internal.cli.main_parser import create_main_parser                                     File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/pip/_internal/cli/main_parser.py", line 5, in <module>                                                                               import subprocess                                                                                File "/data/data/com.termux/files/usr/lib/python3.12/subprocess.py", line 104, in <module>        
   from _posixsubprocess import fork_exec as _fork_exec                                           
ImportError: dlopen failed: cannot locate symbol "_Py_NoneStruct" referenced by "/data/data/com.termux/files/usr/lib/python3.12/lib-dynload/_posixsubprocess.cpython-312.so"...                     
  Setting up python-static (3.12.0) ...                                                              
Setting up python-tkinter (3.12.0) ...                                                             
Setting up python-numpy-static (1.26.1-1) ...                                                      
Processing triggers for man (1.14.6-1) ...

[sylirre]

@Randrianasulu Using wildcards with dpkg won't result in proper installation order of dependencies.

[Randrianasulu]

@Randrianasulu Using wildcards with dpkg won't result in proper installation order of dependencies.

well, I tried to reinstall with pkg reinstall ~/debs/python-ensurepip-wheels_3.12.0_all.deb

and then.python and python-static, and then again ensurepip and python-pip still giving me same error ....

[sylirre]

pkg install ./*.deb is more correct variant as it would pass debs to apt and it would handle correct installation order.

However yes, it looks like python was built with issues.

Importing subprocess module results in error:

~/debs $ python3
Python 3.12.0 (main, Oct 17 2023, 08:53:28) [Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0 on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.12/subprocess.py", line 104, in <module>
    from _posixsubprocess import fork_exec as _fork_exec
ImportError: dlopen failed: cannot locate symbol "_Py_NoneStruct" referenced by "/data/data/com.termux/files/usr/lib/python3.12/lib-dynload/_posixsubprocess.cpython-312.so"...
>>> 

I guess similar errors may appear with other modules that use native extensions.

Also it looks like Python was built without readline support, i.e. cursor moving by arrows and history navigation doesn't work (raw escape sequences appear when using arrows):

>>> print("test^[[D^[[A^[[C^[[B

Basically this PR is unusable.

[landfillbaby]

yeah i kind of abandoned this since python 3.12 drops so much support. someone else can take over if they want

[yubrajbhoi]

Looks like some old patches were not applied. Here are all the patches: https://github.com/yubrajbhoi/termux-python-12/tree/main/patches

I have tested them with Python 3.12.0 and 3.12.1. Every module seems to be working, including subprocess. Also, AFAIK distutils is still available from setuptools package. Can't we use that?

I used that to build Numpy for Python-3.12.1 using the setup.py script and it worked.

[licy183]

It tests fine on my device.

@landfillbaby @sylirre Could you please test it again?

The setuptools should be hardcoded to 67.6.1 if crossenv wants to be used, even 67.7 can't be used due to some weird errors...

[sylirre]

@licy183 Looks good, tested few modules and can't find issues.

[landfillbaby]

did some testing as asked, LGTM ! :) sorry i've not been active recently

[licy183]

Next, we should find all the revdeps of python, check whether it supports python3.12 and revbump if needed.

[codenamedpkt]

distutils still exists and not archived from pypa's GitHub Account, they also belongs to Python.org: https://github.com/pypa/distutils

[licy183]

Revdeps of python

Package Name	Need Rebuild?	Type	Remark
2ping	Yes	Pure Python Package
asciidoc	Yes	Pure Python Package
asciinema	Yes	Pure Python Package
autojump	Yes	Scripts Using Python	Strange installation folder
boost	Yes	Depends on Python	libboost-python
borgbackup	Yes	Python Package
calcurse	Yes	Scripts Using Python	Installs deps in postinst
dbus-python	Yes	Python Package
electrum	Yes	Pure Python Package
emscripten	?	Depends on Python?	Ping @truboxl
fetchmailconf	Yes	Pure Python Package
frida-python	Yes	Python Package
gdb	Yes	Depends on Python
gjots2	Yes	Pure Python Package
glib-bin	No	Scripts Using Python
gnumeric-python	Yes	Python Package
g-ir-scanner	Yes	Depends on Python	_giscanner.cpython-311.so
gst-python	Yes	Python Package
hash-slinger	Yes	Scripts Using Python	Installs deps in postinst
hexchat	Yes	Depends on Python
inkscape	Yes	Scripts Using Python	Installs deps in postinst
iwyu	No	Scripts Using Python
jbig2enc	No	Scripts Using Python
ki18n	No	Scripts Using Python
kitty	Yes	Depends on Python
ledger	Yes	Python Package
lldb	Yes	Depends on Python	Install files to $PYTHONPATH
libncnn	Yes	Depends on Python	Install files to $PYTHONPATH
libpcsclite	Yes	Depends on Python
libplacebo	No?	Don't know	Seems not depending on Python
libsearpc	Yes	Depends on Python	Install files to $PYTHONPATH
libtorrent-rasterbar	Yes	Depends on Python	Install files to $PYTHONPATH
tsduck-python	No?	Depends on Python
pyunbound	Yes	Python Package
libxml2-python	Yes	Python Package
lilypond	Yes	Depends on Python	Has Python bindings
lv2	Yes	Scripts Using Python	Installs deps in postinst
matplotlib	Yes	Python Package
olivia	No?	Don't know	Seems not depending on Python
opencv-python	Yes	Python Package
orca	Yes	Depends on Python	Has Python bindings
panda3d	Yes	Depends on Python
pastebinit	No	Scripts Using Python
pathpicker	No	Scripts Using Python
profanity	Yes	Depends on Python
pyatspi	Yes	Python Package
pycairo	Yes	Python Package
pygobject	Yes	Python Package
pyqt5	Yes	Python Package
python-apsw	Yes	Python Package
python-apt	Yes	Python Package
python-bcrypt	Yes	Python Package
python-contourpy	Yes	Python Package
python-cryptography	Yes	Python Package
python-greenlet	Yes	Python Package
python-grpcio	Yes	Python Package
python-lameenc	Yes	Python Package
python-libsass	Yes	Python Package
python-lxml	Yes	Python Package
python-msgpack	Yes	Python Package
python-numpy	Yes	Python Package
python-pillow	Yes	Python Package
python-pyqtwebengine	Yes	Python Package
python-qscintilla	Yes	Python Package
python-sabyenc3	Yes	Python Package
python-scipy	Yes	Python Package
python-tldp	Yes	Python Package
python-torch	Yes	Python Package
python-torchaudio	Yes	Python Package
python-torchvision	Yes	Python Package
python-xlib	Yes	Python Package
ranger	Yes	Depends on Python	Install files to $PYTHONPATH
rdiff-backup	Yes	Pure Python Package	Installs deps in postinst
rdircd	Yes	Scripts Using Python	Installs deps in postinst
recoll-python	Yes	Python Package
rhythmbox	No	Scripts Using Python
sabnzbd	Yes	Scripts Using Python	Installs deps in postinst
seafile-client	Yes	Depends on Python	Install files to $PYTHONPATH
speechd	Yes	Depends on Python
srt2vobsub	Yes	Scripts Using Python	Installs deps in postinst
termux-apt-repo	No	Scripts Using Python
termux-create-package	No	Scripts Using Python
termux-gui-package	Yes	Scripts Using Python	Installs deps in postinst
termux-gui-pm	Yes	Scripts Using Python	Installs deps in postinst
tizonia	Yes	Depends on Python	Install files to $PYTHONPATH
uftrace	Yes	Depends on Python
units	Yes	Scripts Using Python	Installs deps in postinst
uwsgi	Yes	Depends on Python
vim-gtk	Yes	Depends on Python
vim-python	Yes	Depends on Python
weechat-python-plugin	Yes	Depends on Python
xfce4-panel-profiles	No	Scripts Using Python
zeronet	No?	Scripts Using Python

[TomJo2000]

I'm gonna pull down this PR to my development fork and do some local builds for all of these.

[TomJo2000]

Finally had the time to finish up local build testing.

82/93

Package Name	Builds?
2ping	[x]
asciidoc	[x]
asciinema	[x]
autojump	[x]
boost	[x]
borgbackup	[x]
calcurse	[x]
dbus-python	[x]
electrum	[x]
emscripten	[x]
fetchmail	[x]
frida	[ ]
gdb	[x]
gjots2	[x]
glib-bin	[x]
gnumeric	[x]
gobject-introspection	[x]
gst-python	[x]
hash-slinger	[x]
hexchat	[x]
inkscape	[x]
iwyu	[x]
jbig2enc	[x]
ki18n	[x]
kitty	[ ]
ledger	[x]
libllvm	[x]
libncnn	[x]
libpcsclite	[x]
libplacebo	[x]
libsearpc	[x]
libtorrent-rasterbar	[ ]
libtsduck	[x]
libunbound	[x]
libxml2	[x]
lilypond	[x]
lv2	[x]
matplotlib	[x]
olivia	[x]
opencv	[ ]
orca	[x]
panda3d	[x]
pastebinit	[x]
pathpicker	[x]
profanity	[x]
pyatspi	[x]
pycairo	[x]
pygobject	[x]
pyqt5	[x]
python-apsw	[x]
python-apt	[x]
python-bcrypt	[x]
python-contourpy	[x]
python-cryptography	[x]
python-greenlet	[x]
python-grpcio	[ ]
python-lameenc	[x]
python-libsass	[x]
python-lxml	[x]
python-msgpack	[x]
python-numpy	[x]
python-pillow	[x]
python-pyqtwebengine	[x]
python-qscintilla	[x]
python-sabyenc3	[x]
python-scipy	[ ]
python-tldp	[x]
python-torch	[ ]
python-torchaudio	[ ]
python-torchvision	[ ]
python-xlib	[x]
ranger	[x]
rdiff-backup	[x]
rdircd	[x]
recoll	[x]
rhythmbox	[x]
sabnzbd	[x]
seafile-client	[x]
speechd	[x]
srt2vobsub	[ ]
termux-apt-repo	[x]
termux-create-package	[x]
termux-gui-package	[x]
termux-gui-pm	[x]
tizonia	[ ]
uftrace	[x]
units	[x]
uwsgi	[x]
vim-gtk	[x]
vim-python	[x]
weechat	[x]
xfce4-panel-profiles	[x]
zeronet	[x]

[TomJo2000]

As mentioned above, since xkeyboard-config now depends on python it'll need to be added to the rebuild list.

[truboxl]

emscripten just need the $PREFIX/bin/python3 interpreter to exist, no python library compiling happens

[codenamedpkt]

We should build a host python for python3.12. Ubuntu 22.04 hasn't packaged it yet. Some other repositories such as ppa/deadsnakes may be an alternative in the future, but it hasn't packaged python3.12 yet.

IMO we had better hold it to 3.11 until some other python libraries support it. Bumping to python3.12 will at least break the build process of numpy because python3.12 remove support for distutils but meson-python (which numpy uses for python3.12) hasn't supported cross-compile.

Look on lines from official meson-python GitHub repo, from pyproject.toml:

Requires = [
  'meson >= 1.2.3; python_version >= "3.12"',
  'packaging >= 19.0',
  'pyproject-metadata >= 0.7.1',
]

They gave up numpy and tomli as requirements and deps, in their main branch.

But, numpy needs back meson-python as meson-python do not need numpy as deps

[TomJo2000]

They gave up numpy and tomli as requirements and deps, in their main branch.

I am not sure where the hell you are getting that from. Surely not mesonbuild/meson-python

Which very much still states:

requires = [
  'meson >= 0.63.3; python_version < "3.12"',
  'meson >= 1.2.3; python_version >= "3.12"',
  'packaging >= 19.0',
  'pyproject-metadata >= 0.7.1',
  'tomli >= 1.0.0; python_version < "3.11"',
]

[codenamedpkt]

Look at https://github.com/mesonbuild/meson-python/blob/main/pyproject.toml#L8-L14

They gave up numpy and tomli as requirements and deps, in their main branch.

I am not sure where the hell you are getting that from. Surely not mesonbuild/meson-python

Which very much still states:

requires = [
  'meson >= 0.63.3; python_version < "3.12"',
  'meson >= 1.2.3; python_version >= "3.12"',
  'packaging >= 19.0',
  'pyproject-metadata >= 0.7.1',
  'tomli >= 1.0.0; python_version < "3.11"',
]

https://github.com/mesonbuild/meson-python/blob/master/pyproject.toml#L9

'meson >= 0.63.3; python_version < "3.12"',

This means, if you have meson grater equal than v0.63.3 but less than v1.2.2, you need python 3.11 or earlier. Or only use the latest meson-python tied with python 3.12.4 and tomli is not needed in latest meson-python commits.

[TomJo2000]

Ah okay I read that wrong then.

[codenamedpkt]

As your table:

Finally had the time to finish up local build testing.

93/93

Package Name	Builds?
2ping	[x]
asciidoc	[x]
asciinema	[x]
autojump	[x]
boost	[x]
borgbackup	[x]
calcurse	[x]
dbus-python	[x]
electrum	[x]
emscripten	[x]
fetchmail	[x]
frida	[x]
gdb	[x]
gjots2	[x]
glib-bin	[x]
gnumeric	[x]
gobject-introspection	[x]
gst-python	[x]
hash-slinger	[x]
hexchat	[x]
inkscape	[x]
iwyu	[x]
jbig2enc	[x]
ki18n	[x]
kitty	[x]
ledger	[x]
libllvm	[x]
libncnn	[x]
libpcsclite	[x]
libplacebo	[x]
libsearpc	[x]
libtorrent-rasterbar	[x]
libtsduck	[x]
libunbound	[x]
libxml2	[x]
lilypond	[x]
lv2	[x]
matplotlib	[x]
olivia	[x]
opencv	[x]
orca	[x]
panda3d	[x]
pastebinit	[x]
pathpicker	[x]
profanity	[x]
pyatspi	[x]
pycairo	[x]
pygobject	[x]
pyqt5	[x]
python-apsw	[x]
python-apt	[x]
python-bcrypt	[x]
python-contourpy	[x]
python-cryptography	[x]
python-greenlet	[x]
python-grpcio	[x]
python-lameenc	[x]
python-libsass	[x]
python-lxml	[x]
python-msgpack	[x]
python-numpy	[x]
python-pillow	[x]
python-pyqtwebengine	[x]
python-qscintilla	[x]
python-sabyenc3	[x]
python-scipy	[x]
python-tldp	[x]
python-torch	[x]
python-torchaudio	[x]
python-torchvision	[x]
python-xlib	[x]
ranger	[x]
rdiff-backup	[x]
rdircd	[x]
recoll	[x]
rhythmbox	[x]
sabnzbd	[x]
seafile-client	[x]
speechd	[x]
srt2vobsub	[x]
termux-apt-repo	[x]
termux-create-package	[x]
termux-gui-package	[x]
termux-gui-pm	[x]
tizonia	[x]
uftrace	[x]
units	[x]
uwsgi	[x]
vim-gtk	[x]
vim-python	[x]
weechat	[x]
xfce4-panel-profiles	[x]
zeronet	[x]

If you mean Kitty is a GPU-ed terminal emus thats one: Kitty Terminal Emulater or using pypi from kitty, it is yanked. Others:

• Frida
• LibTorrent as libtorrent-rasterbar
• OpenCV
• gRPCIO
• SciPy
• PyTorch
• Audio by PyTorch
• Vision by PyTorch
• SRT2VobSub
• Skema by Tizonia

I finished up here.

[TomJo2000]

If you mean Kitty is a GPU-ed terminal emus thats one: Kitty Terminal Emulater or using pypi from kitty, it is yanked. Others:

• Frida
• LibTorrent as libtorrent-rasterbar
• OpenCV
• gRPCIO
• SciPy
• PyTorch
• Audio by PyTorch
• Vision by PyTorch
• SRT2VobSub
• Skema by Tizonia

I finished up here.

Those packages are referring to all packages in our repositories that depend on Python. These 11 were the ones that failed in my local build testing. Are you saying you tested them and they build fine? Or are you just listing them for no reason.

[codenamedpkt]

If you mean Kitty is a GPU-ed terminal emus thats one: Kitty Terminal Emulater or using pypi from kitty, it is yanked. Others:

• Frida
• LibTorrent as libtorrent-rasterbar
• OpenCV
• gRPCIO
• SciPy
• PyTorch
• Audio by PyTorch
• Vision by PyTorch
• SRT2VobSub
• Skema by Tizonia

I finished up here.

Those packages are referring to all packages in our repositories that depend on Python. These 11 were the ones that failed in my local build testing. Are you saying you tested them and they build fine? Or are you just listing them for no reason.

They are not depends to actual Python. We just list them for reference.

[codenamedpkt]

If you mean Kitty is a GPU-ed terminal emus thats one: Kitty Terminal Emulater or using pypi from kitty, it is yanked. Others:

• Frida
• LibTorrent as libtorrent-rasterbar
• OpenCV
• gRPCIO
• SciPy
• PyTorch
• Audio by PyTorch
• Vision by PyTorch
• SRT2VobSub
• Skema by Tizonia

I finished up here.

Those packages are referring to all packages in our repositories that depend on Python. These 11 were the ones that failed in my local build testing. Are you saying you tested them and they build fine? Or are you just listing them for no reason.

They are not depends to actual Python. We just list them for reference.

You can use pip install -U --no-cache-dir --force-reinstall --clean your_http_github_or_gitlab_link ... .