libatomic dependency may be problematic

[indygreg]

A dependency to libatomic was added in #400.

The libatomic inclusion might be concerning because it isn't listed in the LSB at https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Common/LSB-Common/requirements.html.

While the LSB may not be the absolute requirements for our distros, deviation from it is concerning and needs to be researched. We can't just introduce new external system dependencies without validating that the most minimal of Linux distros ship the library. If a library is distributed by e.g. the compiler toolchain, that's bad for portability.

I believe I've ran into issues with libatomic in the past. Check the history of the OpenSSL build scripts.

I'd feel much better if libatomic.so linking advertising and the Rust validation is exclusive to MIPS at the moment. And even that may be too lenient. We would need to find a MIPS Linux distro and see where libatomic comes from.

[zanieb]

Thanks for following up here.

I'm not sure what changed upstream that caused the MIPS builds to regress, that seems like the first thing to look into. I'm particularly surprised that libatomic is required by the non-freethreaded variant as well. I've opened an upstream issue https://github.com/python/cpython/issues/127708.

It also sounds very reasonable to scope the validation to MIPS while we investigate that. See https://github.com/indygreg/python-build-standalone/pull/411.

Regarding the choice to allow this in #400, I wanted to prioritize getting the security patches out for the commonly-used distributions. Given the very long CI iteration time, I opted to include the change without further investigation. I'm not sure if you feel differently about how to weigh those decisions, but hopefully there will be less of a trade-off in the future as we invest into reducing build and iteration times — I've already prototyped using paid runners to significantly improve the concurrency of the builds and your work on #392 is promising.

[WilliamStam]

this is causing issues with packaging builds using nuitka on arch and red hat. (ubuntu seems fine)

https://github.com/astral-sh/python-build-standalone/issues/616

(sorry just highlighting this here)

[polarathene]

TL;DR: Interim solution is -l:libatomic.a to -l atomic, and ensure the static lib is found first.

---

Apply your workaround to remove libatomic.so in the container like you do for other .so here:

https://github.com/astral-sh/python-build-standalone/blob/1abe86213664d4a47226779f3cc0b0daaa9b46ab/cpython-unix/build-cpython.sh#L44-L47

This doesn't cover the separate toolchain search paths (hence why -l:libatomic.a was probably chosen as a workaround?).

So just identify that location and strip it too; alternatively symlink/copy your static libs you want to prioritize and add a search path for those (as described here).

EDIT: It seems you already have your own dir for this with ${TOOLS_PATH}/deps/, so symlink your libatomic.a to that if that dir has precedence. If the compiler toolchain is prioritizing it's own search paths still you may need to use some options to adjust that.

This commit (Sep 2023) introduced the -l:libatomic.a for OpenSSL 3.x with the explanation:

Clang ships a dynamic libatomic.so by default. So if we're not careful we get a run-time dependency on libatomic.so. We work around this by using the -l:libatomic.a linker flag syntax when building/linking CPython extensions using libcrypto to force the symbols to get linked statically.

If you have a controlled build environment, you should not have this concern :) (can be easily caught via CI too)

Original Response follows

@WilliamStam your problem is due to whatever decision Nuitka is doing on it's end with the information from python-build-standalone. Seems to be due to originally implementing support for static lib linking, that should no longer be relevant to Nuitka unless I'm missing something on what that project is doing.

---

@indygreg @zanieb

What has not been clarified is why -l atomic is not viable in the meantime. The issue description here does not that adding a runtime dep for libatomic is not desirable, but I'm not quite sure where this would be the case?

If it's concerning python-build-standalone with it's own CI builds, you can still use -l atomic like -l:libatomic.a and benefit from static inclusion in your published builds. This would treat libatomic the same as all others in the sysconfig MODLIBS list of linked libs where there's clearly already implicitly static linked libs.

Downstream projects like Nuitka are not using any of the other link args from this config data, so not only do you keep your own builds with static libatomic, but it fixes the downstream concern too. Nuitka won't attempt to link libatomic as a result, those users are then happy (Nuitka can likewise fix this on their end AFAIK).

Any other project that would use this "leaked" info differently would be treating the other libs all the same already. So I really don't see a problem?