closure depends on the system linker

[guibou]

I had build clotestbin-cc-pie.sh on nixos using:

nix-shell
bazel build //:clotestbin-cc-pie

After coping clotestbin-cc-pie.sh to another computer, I realized that the closure depends on the static linker of the building environment.

For example, after patching clotest-cc-pie.sh so that it displays the list of executed command and does not remove the temporary directory and inside a docker container running an ubuntu

root@4a9668643bbb:/chien# ./clotestbin-cc-pie.sh 
+++ mktemp -d
++ tmpdir=/tmp/tmp.3emtCtrOEB
++ unzip -q ./clotestbin-cc-pie.sh -d /tmp/tmp.3emtCtrOEB
++ true
++ /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
./clotestbin-cc-pie.sh: line 6: /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper: No such file or directory

root@4a9668643bbb:/chien# file /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper 
/tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/l0adcz8y05vajfi483a7qv31i03rspqq-glibc-2.27/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

I'm on the ubuntu system and it tries to load the linker from /nix/store path.

Overriding the static linker (using patchelf) makes it work:

root@4a9668643bbb:/chien# patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper 
root@4a9668643bbb:/chien# /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
hello cc

Please note that's not a nix(os) specific issue, because using the current system dynamic loader may not work. For example, a closure built on a recent (2019) nixos works on a ubuntu 18.04 with overrided dynamic loader. However, on an ubuntu 14.04 it does not work and fails with the following error:

/tmp/tmp.IpokZOrdzx/clotestbin-cc-pie-closure_wrapper: relocation error: /tmp/tmp.IpokZOrdzx/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

One solution to this problem may be to bundle the dynamic loader too and use it to run the closure in the running shell as such:

/tmp/tmp.IpokZOrdzx/ld-linux-x86-64.so.2 /tmp/tmp.IpokZOrdzx/clotestbin-cc-pie-closure_wrapper

[facundominguez]

One solution to this problem may be to bundle the dynamic loader too and use it to run the closure in the running shell as such

This is the approach of exodus. It should work for executables.

[jkachmar]

I've used this process with musl's dynamic loader and it works very nicely for generating portable executables; so anecdotal 👍 from me.

[matthewbauer]

I've used this process with musl's dynamic loader and it works very nicely for generating portable executables; so anecdotal +1 from me.

Meaning you ship the dynamic linker with your binary? The annoying part about this is it means you have to have an absolute path for ld.so to go, since relative paths aren't allowed in the interpreter.

I was hoping there were some flags you could pass to gcc / glibc to enable a compatibility mode for older interpeters. It sounds like the best advice for this is to just to link to a really old glibc :(. Static linking looks like the best option for this kind of portability.

[jkachmar]

Meaning you ship the dynamic linker with your binary? The annoying part about this is it means you have to have an absolute path for ld.so to go, since relative paths aren't allowed in the interpreter.

Yes, what I ended up doing was appending a script to the compressed file that auto-extracted the whole thing to a known location in the system temporary directory so that the linker could be called from a known absolute path.

It mostly worked fine, but there are some drawbacks that need to be worked around (ensuring that files get cleaned up, avoiding gratuitous space leaks, etc.); IIRC exodus does this the "intelligent" way (i.e. it hashes the components and checks to avoid duplicating stuff).

Static linking looks like the best option for this kind of portability.

:100:; sadly easier said than done...

[aherrmann]

Meaning you ship the dynamic linker with your binary? The annoying part about this is it means you have to have an absolute path for ld.so to go, since relative paths aren't allowed in the interpreter.

Yes, what I ended up doing was appending a script to the compressed file that auto-extracted the whole thing to a known location in the system temporary directory so that the linker could be called from a known absolute path.

I might be missing something, but AFAIK you could also invoke the loader directly in your wrapper script and pass your binary as an argument, see ld-linux (8). I.e. something like

$BUNDLE/my-ld-linux.so $BUNDLE/my-binary

You can also use the --library-path argument to point to bundled libraries.

[matthewbauer]

I might be missing something, but AFAIK you could also invoke the loader directly in your wrapper script and pass your binary as an argument, see ld-linux (8). I.e. something like

Good point! It adds some bash wrapping that's a little bit annoying vs. just run the ELF. But as long as you handle args properly, it should be just as good.