Allow dynamic loading in musl builds while CPython itself is static

[achimnol]

trafficstars

I'd like to ship a few wheels including pre-built C modules along with the musl-based standalone Python from here. When I try to run python -m ensurepip (e.g., #84) or any other imports that involves dynamic loading of C libraries (e.g., ctypes, pyzmq, etc.) then it exits with OSError: Dynamic loading not supported.

Could we have a musl-based static build that allows dynamic loading of additional non-standard modules? It seems that 0d3ae2a987ff05f042b4c19b3998f96d79bb03a6 has disabled the ability to use dynamic loading.

[indygreg]

When you produce a fully static ELF executable, you can't load dynamic libraries. That Dynamic loading not supported error is coming from the bowels of libc I believe.

Now, what we probably could do is produce a Python distribution that is dynamically linked against musl libc and the executable only has a dependency on musl. This should enable the distributions to load extensions that were also compiled to target musl libc.

We could potentially also include a placeholder shared library to force the resulting executable to be dynamic. Or maybe there's another way to force the linker into making the final executable dynamic.

Something to keep in mind here is once you support loading dynamic libraries, you have to worry about multiple libc versions being loaded. I believe musl - like glibc - is sensitive about multiple versions existing in the same process space. So we probably don't want to statically link musl and enable extensions to also statically link musl, as this is a recipe for disaster. Having a dynamically linked musl is probably the safest route here.

Out of curiosity, could you please explain your use case? Is this for better support for Alpine Linux and/or compatibility with the new platform tag for musl (https://discuss.python.org/t/pep-656-platform-tag-for-linux-distributions-using-musl/7165)?

[achimnol]

Thanks for the reply. My use case is to mount and execute a Python program with self-contained runtime inside arbitrary Docker containers based on arbitrary images. So far, it's been successful with glibc-flavor of static Python builds and glibc-based Docker images (e.g., Ubuntu/CentOS based images). But for Alpine-based images, a dynamic version of Python build works as expected while no static version did work because of the library import issue.

I think there are two ways if it's not feasible to build static-musl version of Python with import capability:

• Just use a dynamic version of Python which is built with sufficiently low version of Alpine Linux, to guarantee the musl compatibility with as many Alpine-based Docker images as possible.
  • I just wanted to avoid manually checking compatibility with arbitrary images by using the static version.
• Embed all external Python packages into the "standard" library and build with the static Python.

[bigcat88]

@indygreg can I help you anything to make musl dynamic build possible for x64 - aarch64? Familiar with this(not with musl\Alpine itself, but with linkers and compilers), can try clone your repo and do that. If you just point me in right direction. :)

As I understand I need:

1. Find a docker image(or build and publish one with lowest musl version)
2. Change linker(and i think some preprocessor) flags from static to dynamic(what to change will be great to know, but not necessary)
3. Write an GitHub action for that(like you do)

And not related question: my friend and i currently developing an open source project where we will use your python standalone, and we need to include some additional python packages in it, that not providing a binary wheels. We plan to do that in GitHub actions and remove all except install and licenses folder and add your License file to it. Do we need to include some additional info(except a link to your project on a main project readme page). Example: alpha_repack

[andrey18106]

Hi! When will dynamic Python standalone be available on Alpine 3.13+?

[bpcreech]

Just piping* up with another use case:

Using Hatch to run Python matrix tests, for wheels built on Alpine. Hatch uses python-build-standalone for this, plus pip to download wheels.

Here is what it looks like when you try that today. pip fails, just like Joongi said.

I see this patch, patch-pip-static-binary.patch, fixes the site-packages/pip which is bundled with the python-build-standalone distro, but that doesn't help because Python now runs the bundled pip wheel in lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl which does not have that patch.

Note that it's not just code that directly uses ctypes which is broken, but basically any use of pip, including hauling in deps for matrix testing.

One could also upstream patch-pip-static-binary.patch to pip. I think that would fix pip (and thus Hatch's matrix testing would work) but it wouldn't help for code under test which actually uses ctypes (which happens to include my use case anyway).

(*piping up, not pip-ing up, because I can't pip. Sorry, bad pip joke.)