Add dynamically linked musl distributions

[zanieb]

As a consequence of #540 I was playing with these concepts and decided to explore it.

This includes #546 (could be merged separately or together), which separates "static" builds from the "musl" triple specifically in favor of a dedicated build option.

The main implementation downside here is that $ORIGIN doesn't work with DT_NEEDED so we need to use RUNPATH instead, which can cause the wrong library to be loaded if LD_LIBRARY_PATH is set. Given the current builds aren't usable at all, I think this is a fine trade-off. We can explore alternatives in the future, like statically linking just libpython.

Another caveat here: for consistency with the glibc builds, we're changing the "default" musl build to be dynamically linked. This is a breaking change in the release artifacts. The statically linked musl build will include a +static suffix. We could do something for backwards compatibility here, but I think this probably makes sense in the long term. My primary concern is that consumers that combine releases (such as uv) would need to encode this change (e.g., toggle the expectation based on the python-build-standalone version tag).

It's challenging to test changes to the release artifact handling. Regardless of approach, this will need a follow-up to adjust that accordingly.

[zanieb]

I got the x86_64-unknown-linux-musl-noopt build passing on my machine and gave it a quick test in an alpine container and it looks like I didn't package it right.

/test # ldd ./python/install/bin/python
	/lib/ld-musl-x86_64.so.1 (0x7ebb73af0000)
Error loading shared library $ORIGIN/../lib/libpython3.13.so.1.0: No such file or directory (needed by ./python/install/bin/python)
	libc.so => /lib/ld-musl-x86_64.so.1 (0x7ebb73af0000)
Error relocating ./python/install/bin/python: Py_BytesMain: symbol not found

I presumed we just didn't package the .so, but it is there

/ # ls /test/python/install/lib/
Tix8.4.3              libpython3.13.so      pkgconfig             tcl8.6
itcl4.2.4             libpython3.13.so.1.0  python3.13            thread2.8.9
libpython3.13.a       libpython3.so         tcl8                  tk8.6

So I futzed with patchelf

/ # patchelf --set-rpath '/test/python/install/lib' /test/python/install/bin/python
/ # ldd /test/python/install/bin/python
	/lib/ld-musl-x86_64.so.1 (0x79dc36f38000)
Error loading shared library $ORIGIN/../lib/libpython3.13.so.1.0: No such file or directory (needed by /test/python/install/bin/python)
	libc.so => /lib/ld-musl-x86_64.so.1 (0x79dc36f38000)
Error relocating /test/python/install/bin/python: Py_BytesMain: symbol not found

Huh

/ # readelf -d /test/python/install/bin/python

Dynamic section at offset 0x388 contains 21 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [/test/python/install/lib]
 0x0000000000000001 (NEEDED)             Shared library: [$ORIGIN/../lib/libpython3.13.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000c (INIT)               0x401000
 0x000000000000000d (FINI)               0x401148
 0x0000000000000019 (INIT_ARRAY)         0x403e48
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x403e50
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x3ff468
 0x000000006ffffef5 (GNU_HASH)           0x3ff428
 0x0000000000000005 (STRTAB)             0x3fd2e0
 0x0000000000000006 (SYMTAB)             0x3ff338
 0x000000000000000a (STRSZ)              167 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x403fe8
 0x0000000000000002 (PLTRELSZ)           48 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x400470
 0x0000000000000000 (NULL)               0x0

I guess I'll try harder?

/ # patchelf --replace-needed '$ORIGIN/../lib/libpython3.13.so.1.0' '/test/python/install/lib/libpython3.13.so.1.0' /test/pyt
hon/install/bin/python
/ # /test/python/install/bin/python --version
Python 3.13.2

Seems like $ORIGIN probably isn't supported? Need to look into that.

[zanieb]

ORIGIN is supported for RPATH, but seemingly not for NEEDED (ref https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c#n897) — switching to that for now.

[zanieb]

Hm okay something is wrong with Python 3.11 — peachy otherwise!

edit: It's loading the libpython from the system, which is 3.11.

[zanieb]

This is loosely ready for review. I have some minor clean-ups and documentation to do.

[zanieb]

I can look into that. Will require #542.

[zanieb]

I seem to have broken the static builds. I'll continue to iterate on that.

[zanieb]

I'm assuming we were previously implicitly doing this for musl?

I'm not sure why I can't reply to this one, but yes. We were normalizing the triple to gnu.

[zanieb]

Are there any other high-profile users we should check, especially ones we know are using the musl builds? (PyOxidizer?)

I want to ack this, but I don't have a good answer. @ofek comes to mind as an automated consumer outside Astral.

[ofek]

Changing the default to something that actually works is desirable in my opinion. It doesn't matter if there is new naming because in my projects that consume (PyApp & Hatch) there is no ability to use newer non-custom builds without a new release because PyApp has a script that generates the hardcoded source of truth, which Hatch then uses in a script to hardcode its source of truth. So I would just have to change the former script.

[zanieb]

I considered a transition plan like

• Continue to publish static builds as we are today
• Create a duplicate archive with +static
• Publish dynamic builds with +shared
• Switch the default to the +shared variant, retaining a +shared duplicate archive
• Drop the +shared archive

However, users still need to encode a specific version of python-build-standalone in which the default changed. After thinking about it more, I'm still leaning towards just changing the defaut.

[geofft]

@ofek by "actually works" I assume you mean "able to dynamically load stuff"?

Then I think we should ship this and call it a bugfix and make sure we ship it at a time that people will be around on Discord etc. to help downstream folks that are broken by the change if any are actually broken by it.

We could also figure out how to make a static musl include a working libdl, which is probably easier than it would be for glibc because musl is generally less complex, but would still be a little bit of a research project.

[zanieb]

The diff is not too complicated, so I'll just merge this and close #546