`PYTHON.json` object file annotations for 3.11 aren't aware of frozen module changes

[indygreg]

Python 3.11 introduced a multi-phase build approach where some additional .pyc are frozen into the binary. The build system changes appear to have invalidated some of our object file annotation assumptions. This is causing linker errors with the 3.11 distributions on PyOxidizer. e.g.

[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(pylifecycle.o): in function `finalize_interp_clear':
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           pylifecycle.c:(.text+0x1cd9): undefined reference to `_Py_Deepfreeze_Fini'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(pylifecycle.o): in function `pycore_interp_init':
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           pylifecycle.c:(.text+0x21c6): undefined reference to `_Py_Deepfreeze_Init'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x18): undefined reference to `_Py_get_importlib__bootstrap_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x38): undefined reference to `_Py_get_importlib__bootstrap_external_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x58): undefined reference to `_Py_get_zipimport_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x98): undefined reference to `_Py_get_abc_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0xb8): undefined reference to `_Py_get_codecs_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0xd8): undefined reference to `_Py_get_io_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0xf8): undefined reference to `_Py_get__collections_abc_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x118): undefined reference to `_Py_get__sitebuiltins_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x138): undefined reference to `_Py_get_genericpath_toplevel'
[2023-01-16T17:07:14Z WARN  pyoxidizer::project_building]           /usr/bin/ld: /home/runner/work/PyOxidizer/PyOxidizer/target/debug/tempdir/pyoxidizerYN643l/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyo3_ffi-a5bfc332461a2de5.rlib(frozen.o):(.data.rel.ro+0x158): undefined reference to `_Py_get_ntpath_toplevel'

I want to say this regression is only limited to the PYTHON.json build metadata on the 3.11 distributions. So I'm not going to make it a release blocker. But it does prevent the adoption of 3.11 distributions in PyOxidizer until we have a fix.

[ofek]

Is there a fix?

[GPla]

Any updates to this?

[ntamas]

I managed to compile pyoxy embedding Python 3.11 on macOS with the following modifications to cpython-3.11.8+20240224-aarch64-apple-darwin-pgo-full.tar.zst:

1. I used ar -xv libpython3.11 deepfreeze.o to extract the missing deepfreeze.o file from the static library (found in python/install/lib)
2. I moved deepfreeze.o to python/build/Python/deepfreeze/deepfreeze.o where it should be based on a vanilla compilation of the official Python 3.11.8 release on macOS.
3. I extended build_info.core.objs with a new string: "build/Python/deepfreeze/deepfreeze.o".

These modifications coupled with a few more changes in https://github.com/ntamas/PyOxidizer/commit/8fc91c4461f5bec0b506cb61ecac6a2a075b3d29 made it possible to build PyOxy. I haven't tested it extensively yet, but it seems like all that python-build-standalone needs to do is to include the missing deepfreeze.o object in the published files and update PYTHON.json to refer to it, as outlined above.

[ntamas]

Related: it seems like the same issue also affects Python 3.12. Furthermore, PYTHON.json for Python 3.12 mistakenly lists Hacl_Hash_SHA2 as a system library. I'll try to file a separate PR to fix this as well.

[ntamas]

#312 filed to fix additional issues in PYTHON.json for Python 3.12. It is best to review and merge #311 and #312 together.