indygreg
libatomic.a dependency causes issues when building on Arch Linux
Nuitka (and probably other py-to-exe software) tries to use libatomic.a instead of libatomic.so, which crashes gcc or other compilers.
In Arch Linux, libatomic.a does not exist anymore.
replacing compiler flags from -l:libatomic.a to -latomic fixes the issue, the dynamic version of libatomic is always available.
Example error:
gcc -o /home/csd4ni3l/Documents/Python3/shatterstack/test.dist/test -s -z noexecstack -export-dynamic -Wl,-R,'$ORIGIN' -Wl,--disable-new-dtags @"./@link_input.txt" -L/home/csd4ni3l/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib -ldl -lm -lpython3.11 -ldl -lpthread -lutil -lrt -lm -l:libatomic.a -l:libatomic.a
/usr/bin/ld: cannot find -l:libatomic.a: No such file or directory
/usr/bin/ld: cannot find -l:libatomic.a: No such file or directory
collect2: error: ld returned 1 exit status
I think this is leaking out from our internal use of -l:libatomic.a (see cpython-unix/extension-modules.yml), the original purpose of which was to avoid needing libatomic on end user systems, but unfortunately the Python extension build process leaks flags from how Python itself was built, which is probably why you're getting that on your build command line.
We have since started also using libatomic.so on some platforms; see #410. It may be the case that we can actually assume there's a platform libatomic.so everywhere.
Alternatively, we should be able to stop leaking this command line flag.
While we could add filtering for this to sysconfig, I don't think it's necessarily reasonable for us to remove every flag we use.
It seems like Nuitka should provide a way to filter what it pulls from sysconfig?
getting this on redhat as well.
is this a dependency for UV? (using nuitka 2.7.16). tried installing sudo dnf install libatomic and running nuitka with the flags
env LDFLAGS="-L/usr/lib/gcc/x86_64-redhat-linux/8/32/libatomic.a -latomic"
run: |
env LDFLAGS="-L/usr/lib/gcc/x86_64-redhat-linux/8/32/libatomic.a -latomic" uv run python -m nuitka src/main.py \
--standalone \
--onefile \
--msvc=latest \
--output-filename=argus-redhat \
--output-dir=build
but no such luck. which project is actually responsible for it? anyone found a way past it?
I might be wrong, but I believe at this time Nuitka works with the Standalone Python Build, albeit in a non-optimal way. My exchange with UV developers has revealed that the static link libraries it contains are tied to the specific compiler that is however not shipped, while Nuitka expects to be able to use any clang or gcc with them. Hence --static-libpython=auto defaults to --static-libpython=no with that Python flavor. Using a managed Python, e.g. Debian/Ubuntu Pythons gives a better result, so does Anaconda, and probably a lot more. This is of course no issue on Windows, where static linking of Python is not possible.
just fired up an ubuntu VM and tried in there. can confirm it works great after installing patchelf
FATAL: Error, standalone mode on Linux requires 'patchelf' to be installed. Use 'apt/dnf/yum install patchelf' first.
after sudo apt install patchelf and voila..
steps:
install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
get python
uv python install
setup venv
uv venv
run build.sh
uv sync
uv build
uv run python -m nuitka src/main.py \
--standalone \
--onefile \
--output-filename=testing \
--output-dir=build
worked perfectly (ubuntu server 24.04)
the same on red hat (8.9) tho fails with
/usr/bin/ld: cannot find -l:libatomic.a
/usr/bin/ld: cannot find -l:libatomic.a
collect2: error: ld returned 1 exit status
scons: *** [/home/munsys_william__bk/testing/build/main.dist/argus.bin] Error 1
FATAL: Failed unexpectedly in Scons C backend compilation.
Nuitka:WARNING: Complex topic! More information can be found at https://nuitka.net/info/scons-backend-failure.html
Nuitka-Reports: Compilation crash report written to file 'nuitka-crash-report.xml'.
after trying to pinpoint if the problem is nuitka or uv side and this thread https://github.com/Nuitka/Nuitka/issues/3474 i went with trying it on vanila python. im copying the results here
can confirm that in my case (red hat 8.9) using uv python leads to the error /usr/bin/ld: cannot find -l:libatomic.a but if you install python normally (or in my case used the old install - python3.6) it works fine.
$ python3 --version
Python 3.6.8
$ source .venv/bin/activate
$ python --version
Python 3.6.8
$ pip install nuitka
...
$ python -m nuitka src/main.py --standalone --onefile --output-filename=testing --output-dir=build
...
$ ll build
total 16328
drwxr-xr-x 3 4096 Oct 1 11:26 main.build
drwxr-xr-x 2 4096 Oct 1 11:26 main.dist
drwxr-xr-x 3 166 Oct 1 11:26 main.onefile-build
-rwxr-xr-x 1 12700058 Oct 1 11:26 testing <--- YES!
Self compiled Python of course can make the same mistakes as the standalone Python build there, and the solution would be the same. Build it correctly or use a supported Python. For Fedora Python I believe that static libpython also defaults to "no" because they don't know how to provide a usable libpython either. Debian and others can do it. Although for Ubuntu I recently needed to give up with Python 3.14, or so it seems. These things are typically very neglected. I might describe more publically how to achieve it, but I lack the time and am out of here for now.
The libatomic dependency is coming from OpenSSL. We could potentially change our OpenSSL build script to statically link libatomic. And then if libssl.a / libssl.so (or libcrypto) have the atomic symbols, we should be able to purge references to libatomic from CPython's build system, eliminating the possibility that sysconfig will leak implementation details.
Assuming CPython itself doesn't direct depend on the symbols in libatomic, removing all references to libatomic from sysconfig could also work.
Short and sweet: Honestly quick and easy fix in this scenario (should libatomic.a not be acceptable, but libatomic.so is), just switch to -l atomic and ensure preference in your container build still selects the libatomic.a, matching the same behaviour as the other link args?
Read on if that was not clear, I think I've covered everything relevant?
This comment follows after a similar one I posted at Nuitka on the topic: https://github.com/Nuitka/Nuitka/issues/3331#issuecomment-3392793550
eliminating the possibility that
sysconfigwill leak implementation details.
I am perhaps lacking the expertise here, is the only concern the presence of -l:libatomic.a?
- I assume other libs there are also statically linked, but by not enforcing static/dynamic linking any build environment that would care about these link args is free to prioritize static libs ahead of dynamic libs?
- If static libs are not relevant in this sysconfig metadata, shouldn't they be filtered out? (including those that are not explicitly static but still linked in statically?)
I'm a tad confused to what exactly is leaked in this scenario (I understand the discussion is focused on -l:libatomic.a, but that aside the two parties involved it is unclear why this is considered a leak given how both have implemented this).
-
python-build-standalonehas a workaround to prioritize static libs, but for some reason has chosen to have an explicitlibatomic.a - Projects like Nuitka are reading the sysconfig MODLIBS and only filtering for the explicitly static linked libs. This seems questionable on their end, and they could just as easily fix it there too if that's the case.
"Leaked" flags concern
unfortunately the Python extension build process leaks flags from how Python itself was built, which is probably why you're getting that on your build command line.
I don't think it's necessarily reasonable for us to remove every flag we use.
The flags in sysconfig MODLIBS entry is only really relevant for linking a new .so of libpython (or to some degree creating a .a static lib), so I don't think there's any need to hide any of that through filtering/removal? 🤷♂️ (NOTE: I don't know much about the Python ecosystem, it would be good to clarify relevance 😅)
There is a bit of duplicates listed, so filtering that to unique items might be nice at least?:
-lbz2 -lffi -ldl -lncursesw -lpanelw -lncursesw -ldb -lmpdec -lexpat -lcrypto -l:libatomic.a -llzma -lrt -lsqlite3 -lssl -lcrypto -l:libatomic.a -luuid -lm -lm -lexpat -ledit -lncursesw -lz
Inspecting symbols in the lib sure enough confirms the libs being statically linked:
$ nm --defined-only --no-weak /root/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/libpython3.10.so.1.0 | less
# Dynamically linked libs:
$ patchelf --print-needed /root/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/libpython3.10.so.1.0
libdl.so.2
librt.so.1
libm.so.6
libpthread.so.0
libutil.so.1
libc.so.6
NOTE: Some users at https://github.com/astral-sh/python-build-standalone/issues/275 are querying about the size of the library, either unaware of all the extra deps linked statically or stating the bundling is a bit more excessive and requesting a minimal version (Ubuntu packages a minimal Python variant for example).
Presently only reported as a problem for Nuitka?
The Nuitka maintainer @kayhayen is interested in a libpython.a instead of a libpython.so output.
The above indicates what deps were linked in, but is only explicit about a static link for libatomic, yet a good portion of those deps appear to be static linked into the library itself, so the build system either lacked dynamic lib copies, or the static libs were in a search path with higher precedence than the .so equivalents.
IIRC I saw in the python-build-standalone source there was some logic to remove the .so equivalents for most of those listed above from system lib paths, as a way to get the linker to favor the static libs, and justifying that as an acceptable workaround because it was a container build.
When in control of how the link args are invoked, you could enforce static lib selection via -Bstatic and switch back to dynamic libs with -Bdyamic.
Alternatively just add "-l:lib${LIB_NAME}.a" (easy enough transform to do), and you'll also prefer static lib selection if you prefer that over the -B flags. If taking the latter approach you can likewise specifically link libs dynamically by using .so instead of .a. Otherwise place your custom built static libs in a new search path and give that priority (preferred IMO).
Nuitka to my knowledge presently mistakenly reads this Python's build sysconfig MODLIBS to carry over any .a links it finds into link args for the output binary it produces (this is handled in Nuitka source via getModuleLinkerLibs(), see "Investigation" section of this comment for more info).
Seems unnecessary though? Nuitka seems to have done this because at the time they had come across a python-build-standalone supplied libpython.a (3.7) rather than a .so. Since creating a static lib would only bundle any built .o objects and defers linking, I'm not sure why only explicit static libs were of interest for Nuitka to capture? 🤷♂️
Drop the explicit static link?
I'm sure I'm missing something... otherwise if Nuitka is still carrying support around a libpython.a assumption, it could instead autodetect that easily enough for -lpython3.10 (ambiguous preference). In Nuitka's case, this is linking to the output binary, where libpython.so shouldn't need any of it's deps carried over AFAIK, it is only doing so for the libpython.a scenario? (where static libs like libatomic could technically be merged IIRC, but -l atomic should work just as well at Nuitka's side?)
Here's an example of preferring ambiguous link args when that makes sense (otherwise explicitly static/dynamic linking as demonstrated here):
$ docker run --rm -it fedora:42 bash
$ dnf install -yq gcc libatomic libatomic-static
$ mkdir /tmp/prefer-static
$ ln --symbolic "$(gcc -print-search-dirs | grep ^install: | cut -d' ' -f2)libatomic.a" /tmp/prefer-static/libatomic.a
$ gcc -shared -nodefaultlibs -L /tmp/prefer-static/ -l atomic -o /tmp/example
# No dynamic link, `/tmp/prefer-static` only contained a `libatomic.a` and was found first:
$ ldd /tmp/example
statically linked
Extra reference context (collapsed for brevity)
# With default glibc link by GCC:
$ gcc -shared -L /tmp/prefer-static/ -l atomic -o /tmp/example
$ ldd /tmp/example
linux-vdso.so.1 (0x00007ffe51715000)
libc.so.6 => /lib64/libc.so.6 (0x0000773962f12000)
/lib64/ld-linux-x86-64.so.2 (0x000077396310d000)
$ gcc -shared -nodefaultlibs -l atomic -o /tmp/example
$ ldd /tmp/example
linux-vdso.so.1 (0x00007ffc57aa8000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x0000712712c7a000)
libc.so.6 => /lib64/libc.so.6 (0x0000712712a88000)
/lib64/ld-linux-x86-64.so.2 (0x0000712712c8e000)
$ patchelf --print-needed /tmp/example
libatomic.so.1
$ gcc -shared -l atomic -o /tmp/example
$ patchelf --print-needed /tmp/example
libatomic.so.1
libc.so.6
Static vs Dynamic choice
@kayhayen since you're linking with Nuitka, you should have the choice here how Nuitka passes these link args to the compiler/linker. Normalize -l:lib<name>.a to -l <name> and problem solved? A user could still link libs statically if they wanted.
My exchange with UV developers has revealed that the static link libraries it contains are tied to the specific compiler that is however not shipped, while Nuitka expects to be able to use any clang or gcc with them.
libatomic is a system library. If you were to link dynamically, GCC provides libatomic.so which is just a linker script to the system versioned library (eg: /usr/lib64/libatomic.so.1.2.0) which has the common DT_SONAME entry of libatomic.so.1, which is the actual lib linked and queried for at runtime (the .so is purely a placeholder for dev convenience at build/link-time).
On Fedora Clang depends on GCC and implicitly includes the GCC search paths, so it will also find libatomic.so at link-time. Other compilers like Zig would need to be given the extra search dir via -L. Same scenario for libatomic.a (via libatomic-static package).
This shouldn't be any different to linking libgcc_s.so.1? (something Rust built software commonly links to, but is not present if compiling with Zig instead)
For Fedora Python I believe that static libpython also defaults to "no" because they don't know how to provide a usable libpython either.
A shared library is still usable, a static lib can be convenient/beneficial but is not always available in a build environment (if depending only on system packages to source that static lib).
Rust *-sys crates often offer support for static builds, but you can look at the issues open on a few of those to see the various headaches that users can run into from offering such, whereas a shared library is much more accessible for them to get a successful build.
I'm not familiar enough with building Python, but the following may be of interest to you:
- https://src.fedoraproject.org/rpms/python3.13/tree/rawhide (the
.specbuilds with--without-static-libpython) - https://fedoraproject.org/wiki/User:Cstratak/python3_static_link_against_libpython (2019 proposal to build static lib, but not distribute it as a package to users only for improving the internal build of the
.so)
So they probably do know how to provide what you'd like, but for whatever reason Fedora doesn't offer a python-static package or similar. Debian/Ubuntu might make you happy in this situation, but they're not without their own caveats in my experiences with using those distros 😓
