Move to manylinux_2_28 for aarch64, ppc64le builds.

[bryanculver]

Related to:

• https://github.com/psycopg/psycopg2/issues/1396 (hopefully getting newer libpq binaries)
• https://github.com/pypa/manylinux/issues/1332 (2_24 being EOL?)
• https://github.com/mayeut/pep600_compliance#distro-compatibility (2_28 compatible base to Debian 10)
• https://github.com/pypa/manylinux/commit/26ca99408eb9d728f355894613d02b28d9d28d1d (2_28 just being added to the build pipeline for manylinux)

I believe this should be the only thing necessary to get to Debian 10 (Buster)

For what it's worth this MAY be able to collapse the CentOS builds from manylinux2014 to manylinux_2_28 as per the compatibility chart above states it supports a minimum of CentOS 8, although I don't want to trigger pitchforks: https://github.com/pypa/manylinux/issues/1332#issuecomment-1148055207

---

I also may be completely missing the issue here. 🤷 Feel free to decline with fervor.

[dvarrazzo]

Oh, is 2_28 already out? Sweet. From https://github.com/pypa/manylinux/issues/1332 I understood it's not out yet.

Yes, it should address #1396 indeed.

[dvarrazzo]

Do I understand correctly that 2_28 is based on Centos rather than on Debian? If so maybe some build script should be adapted.

Are you able to create a test build of these packages? Thank you!

[bryanculver]

Do I understand correctly that 2_28 is based on Centos rather than on Debian? If so maybe some build script should be adapted.

Are you able to create a test build of these packages? Thank you!

Valid point! I erroneously assumed it was Debian based but I should double check how to get installed. I’ll try to run this locally later today.

[bryanculver]

😬 https://github.com/pypa/manylinux/issues/1332#issuecomment-1148206050

I’ll see how easy this is to get to work but might be a little early with this PR.

🤞

[bryanculver]

OK so I pushed an update which install libpq-devel properly. And to my surprise... it works! (I think)

There are some hiccups though when it comes to post-build steps and maturity of manylinux_2_28 so please continue on after this output log for some of my findings.

[root@746bbcededce psycopg2]# yum info libpq-devel
Last metadata expiration check: 1:11:53 ago on Tue 07 Jun 2022 12:56:30 PM UTC.
Installed Packages
Name         : libpq-devel
Version      : 13.5
Release      : 1.el8
Architecture : x86_64
Size         : 324 k
Source       : libpq-13.5-1.el8.src.rpm
Repository   : @System
From repo    : appstream
Summary      : Development files for building PostgreSQL client tools
URL          : http://www.postgresql.org/
License      : PostgreSQL
Description  : The libpq package provides the essential shared library for any PostgreSQL
             : client program or interface.  You will need to install this package to build any
             : package or any clients that need to connect to a PostgreSQL server.

Available Packages
Name         : libpq-devel
Version      : 13.5
Release      : 1.el8
Architecture : i686
Size         : 98 k
Source       : libpq-13.5-1.el8.src.rpm
Repository   : appstream
Summary      : Development files for building PostgreSQL client tools
URL          : http://www.postgresql.org/
License      : PostgreSQL
Description  : The libpq package provides the essential shared library for any PostgreSQL
             : client program or interface.  You will need to install this package to build any
             : package or any clients that need to connect to a PostgreSQL server.




[root@746bbcededce psycopg2]# ./scripts/build/build_manylinux_2_28.sh
+++ dirname ./scripts/build/build_manylinux_2_28.sh
++ cd ./scripts/build
++ pwd
+ dir=/app/psycopg2/scripts/build
++ cd /app/psycopg2/scripts/build/../..
++ pwd
+ prjdir=/app/psycopg2
+ '[' '!' '' ']'
++ ls /opt/python/
+ PYVERS='cp310-cp310
cp36-cp36m
cp37-cp37m
cp38-cp38
cp39-cp39
pp37-pypy37_pp73
pp38-pypy38_pp73
pp39-pypy39_pp73'
++ grep -e '^PSYCOPG_VERSION' /app/psycopg2/setup.py
++ sed 's/.*'\''\(.*\)'\''/\1/'
+ version=2.9.3
+ distdir=/app/psycopg2/dist/psycopg2-2.9.3
+ [[ -n '' ]]
+ yum install -y libpq-devel
Last metadata expiration check: 0:03:33 ago on Tue 07 Jun 2022 12:56:30 PM UTC.
Package libpq-devel-13.5-1.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ for pyver in $PYVERS
+ pybin=/opt/python/cp310-cp310/bin
+ /opt/python/cp310-cp310/bin/pip wheel /app/psycopg2 -w /app/psycopg2/dist/
Processing /app/psycopg2
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl size=490212 sha256=06a8cc19cc9ea2b80de2ccbbeb4796bd063267e1079ff4bc93130419a0c7942d
  Stored in directory: /tmp/pip-ephem-wheel-cache-v_6xy8w6/wheels/97/0b/f4/b1e62973ad4dbf6950a3847822f9a8914263514139f5b3b069
Successfully built psycopg2
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/opt/_internal/cpython-3.10.4/bin/python -m pip install --upgrade pip' command.
+ for pyver in $PYVERS
+ pybin=/opt/python/cp36-cp36m/bin
+ /opt/python/cp36-cp36m/bin/pip wheel /app/psycopg2 -w /app/psycopg2/dist/
Processing /app/psycopg2
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.9.3-cp36-cp36m-linux_x86_64.whl size=459292 sha256=f21fb7152b4ad7978cc1350b098c381a0cf313d4a4183ea180d2ef6b43ab7379
  Stored in directory: /tmp/pip-ephem-wheel-cache-22a1au3p/wheels/87/3a/7c/5fa620050a23f8f66753c5ab5df28a023f7f4375d559cc65f9
Successfully built psycopg2
###############################################
#👉..(continues for all available Python versions. I do need to rerun for -binary)
#👉..(Haven't spun up with Postgres running to run tests but could try to get that to work)
#👉..(Post-build steps fail though, please see below)
###############################################




[root@746bbcededce psycopg2]# ls /opt/python/
cp310-cp310  cp36-cp36m  cp37-cp37m  cp38-cp38	cp39-cp39  pp37-pypy37_pp73  pp38-pypy38_pp73  pp39-pypy39_pp73


[root@746bbcededce psycopg2]# /opt/python/cp310-cp310/bin/python -c "import psycopg2; print(psycopg2.__libpq_version__)"
130005 # 🎉



[root@746bbcededce psycopg2]# uname -r
5.10.104-linuxkit

[root@746bbcededce psycopg2]# cat /etc/*release
AlmaLinux release 8.5 (Arctic Sphynx)
AlmaLinux release 8.5 (Arctic Sphynx)
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.5"

AlmaLinux release 8.5 (Arctic Sphynx)
AlmaLinux release 8.5 (Arctic Sphynx)

---

The build script doesn't get down to testing because it fails to run the strip_wheels script because there is no default python executable it seems. Was curious your thoughts (or maybe this has been brought up elsewhere) how to use/specify a default Python version in manylinux distros:

[root@746bbcededce psycopg2]# ./scripts/build/build_manylinux_2_28.sh
#...END OF RUNNING BUILD SCRIPT ABOVE
+ wheel=/app/psycopg2/dist/psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl
+ shift
+ which python
+ alias
+ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot python
+ eval declare -f
++ declare -f
/usr/bin/which: no python in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
+ py=python3
++ mktemp -d
+ tmpdir=/tmp/tmp.BI2r8nv47J
+ trap 'rm -r /tmp/tmp.BI2r8nv47J' EXIT
+ cd /tmp/tmp.BI2r8nv47J
+ python3 -m zipfile -e /app/psycopg2/dist/psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl .
/app/psycopg2/scripts/build/strip_wheel.sh: line 33: python3: command not found
+ rm -r /tmp/tmp.BI2r8nv47J

[root@746bbcededce psycopg2]# which python
/usr/bin/which: no python in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
[root@746bbcededce psycopg2]# which python3
/usr/bin/which: no python3 in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

---

~Another footnote: It appears manylinux_2_28 is still marked as POC/validation, I had use the :poc label:~

CONTAINER ID   IMAGE                                        COMMAND                  CREATED        STATUS        PORTS                              NAMES
746bbcededce   quay.io/pypa/manylinux_2_28_poc_x86_64:poc   "manylinux-entrypoin…"   2 hours ~ago    Up 2 hours                                       manylinux

~See: https://github.com/pypa/manylinux/issues/1282#issuecomment-1037985255~

EDIT: Looked at the wrong image name. There is a manylinux_2_28 tagged latest: https://quay.io/repository/pypa/manylinux_2_28_x86_64?tab=tags&tag=latest

[dvarrazzo]

Source       : libpq-13.5-1.el8.src.rpm

is libpq 14 not available? It's an important version as it adds the pipeline mode.

Also, can you check what libssl is installed?

Thank you so far!

[bryanculver]

Source       : libpq-13.5-1.el8.src.rpm

is libpq 14 not available? It's an important version as it adds the pipeline mode.

Also, can you check what libssl is installed?

Thank you so far!

Not out of the box but I can check how hard it would be to get a newer version. I’ll check the version of libssl too.

[bryanculver]

[root@682be270549f psycopg2]# yum list installed | grep ssl
openssl.x86_64                          1:1.1.1k-6.el8_5                        @baseos
openssl-libs.x86_64                     1:1.1.1k-6.el8_5                        @System
[root@682be270549f psycopg2]# yum info openssl-libs
Last metadata expiration check: 3:11:30 ago on Tue 07 Jun 2022 08:44:34 PM UTC.
Installed Packages
Name         : openssl-libs
Epoch        : 1
Version      : 1.1.1k
Release      : 6.el8_5
Architecture : x86_64
Size         : 3.6 M
Source       : openssl-1.1.1k-6.el8_5.src.rpm
Repository   : @System
Summary      : A general purpose cryptography library with TLS implementation
URL          : http://www.openssl.org/
License      : OpenSSL and ASL 2.0
Description  : OpenSSL is a toolkit for supporting cryptography. The openssl-libs
             : package contains the libraries that are used by various applications which
             : support cryptographic algorithms and protocols.

[bryanculver]

@dvarrazzo Can you take a look at any of the feedback above to see if I should update my commits here?

Also I assume #1482 is holding up this getting merged?

[h-vetinari]

Also I assume #1482 is holding up this getting merged?

How are these related? Manylinux wheels for aarch and osx-arm wheels have no overlap.

[bryanculver]

Also I assume #1482 is holding up this getting merged?

How are these related? Manylinux wheels for aarch and osx-arm wheels have no overlap.

~I was unaware #1482 was exclusive osx-arm wheels and curious to know where aarch wouldn't be sufficient.~

[dvarrazzo]

Hello,

I can't remember what I did for psycopg 3, but probably I will adopt something similar, involving building libpq from source and using manylinux2014 tag for maximum compatibility.

[bryanculver]

Also I assume #1482 is holding up this getting merged?

How are these related? Manylinux wheels for aarch and osx-arm wheels have no overlap.

~I was unaware #1482 was exclusive osx-arm wheels and curious to know where aarch wouldn't be sufficient.~

My mistake @h-vetinari. For some reason I had believed that manylinux aarch was a listed compatibility tag for native macOS execution but I just checked again and it is in fact not listed.

[bryanculver]

Hello,

I can't remember what I did for psycopg 3, but probably I will adopt something similar, involving building libpq from source and using manylinux2014 tag for maximum compatibility.

In that case would you like me to close this PR or attempt to port that work from psycopg 3?

[dvarrazzo]

If you would like to work on it, sure

[gilbahat]

Hi, it's been 3 months now from the last comment and the PR is still open. #1396 has been open for almost a year now, causing major pain for anyone trying to migrate to aarch64. can we please merge this?

[dvarrazzo]

@gilbahat no, not in this form, as per last comments. We cannot use libpq from packages on centos, we need to build it from source.

[ddelange]

fwiw, pg v15 is available for pypa's manylinux_2_28 image (Almalinux 8, RHEL8.6) from pg's official 3rd party yum repo.

For asyncpg (PR), however, I had some troubles installing it due to some GPG key issue which should've been fixed but somehow I still encountered it, so I ended up using using pg v13 from the vanilla yum repo. I don't have experience with yum though. Maybe it runs by now, or someone else knows the fix, either way I thought it worth mentioning :)

[dvarrazzo]

@ddelange thank you for the heads-up. However, the main problem with yum packages, is that the distro uses an outdated version of libssl. As a consequence we have to build it from source, and build the libpq as a consequence.

Psycopg 3 does this (hopefully) right, so, the best thing to do is to use the psycopg 3 script for psycopg 2 too.

[ddelange]

thanks for the quick reply! makes sense :) cc @elprans

[elprans]

asyncpg does not have this problem, because it does not link to any libraries aside from libc