debian-stretch build broken (by asm header hackery in libsystrap)

[stephenrkell]

(Reported by @clearyf -- with thanks!)

Building the debian-stretch Dockerfile on buster systems is broken as struct ucontext & ucontext_t have changed between the two releases.

[stephenrkell]

I was hoping that the use of Docker would isolate the guest build environment from the vagaries of the host. Or do you mean you're using the Dockerfile as an informal recipe for building directly in a buster host system?

I have a memory of fixing the ucontext_t problem, but that might simply be a related problem. Will investigate further.

[clearyf]

After having another look, the problem is not with my host system. I found this qemu patch that was required to build with the newer glibc, I came across this in the context of building older qemu on current glibc, so here's the description:

Subject: [PATCH] Replace 'struct ucontext' with 'ucontext_t' type

glibc used to have:

   typedef struct ucontext { ... } ucontext_t;

glibc now has:

   typedef struct ucontext_t { ... } ucontext_t;

(See https://sourceware.org/bugzilla/show_bug.cgi?id=21457
 for detail and rationale for the glibc change)

However, QEMU used "struct ucontext" in declarations. This is a
private name and compatibility cannot be guaranteed. Switch to
only using the standardized type name.

So for me what failed was docker build -t liballocs_built liballocs/buildtest/debian-stretch, but replacing stretch with buster worked fine (modulo issue #32). I'll have to rebuild to get the actual error message, but my guess currently is that one of the liballocs dependencies in contrib tries to use struct ucontext_t, but that isn't a valid type in stretch's glibc. If it used struct ucontext then the build would work on stretch, but fail on buster, if it used ucontext_t then it would work on both, but struct ucontext_t only works on newer glibc and anybody who tries to use anything but the latest & greatest is going to have problems. Once I can sit down for a while with a power supply I'll figure out what is actually causing the problem and maybe we can get it fixed upstream.

[clearyf]

I think the title needs to change, struct ucontext is present in a sense but it's different to what I wrote earlier on, so the basic problem is that both generic asm includes & x86_64 includes are getting mixed up and the build fails due to structs being redefined. I'm not an expert on these details, but I have attached a build log, it should help.

liballocs-stretch-build-failure.txt

[stephenrkell]

Ah yes, OK. Thanks for this.

Yes, there is a mess in libsystrap around raw syscalls and including the right headers for these. I might be wrong, but my tentative conclusion from trying (a while back) was that there is no way to get the kernel-level C struct definitions in portable code... because includes of asm headers will pull in libc headers unintentionally. (It's possible that this can be avoided, but I have so far failed to figure it out.)

My tentative solution is to extract raw syscalls' struct definitions from kernel DWARF directly. I already have some code for this (in my dwarfidl repository -- see 'dwarfhpp') but it's incidentally broken by some annoying things the kernel does that effectively obfuscate the syscall entry-point. It's not too hard for my tool to de-obfuscate. but I haven't done that yet.

In the meantime, perhaps the right thing is to remove the stretch Dockerfile, which by now is rather outdated anyway? The immediate pain comes from libsystrap not being able to hack around stretch-era and buster-era libc headers simultaneously.

[clearyf]

Given what you wrote I'd remove it, although it might be painful for some, buster has only been out since July, but it really depends on the timeframe for the extraction of the syscall struct definitions that you described. If it's anything more than a few months max then by the time that works stretch will be fading into memory for many. The group of people who are interested in working with liballocs at the moment probably don't overlap too well with the group who stay using the same old version of Debian for years on end because they need the stability for whatever reason (ironically I'm in this position where I work as we need a stable base for yocto to work reliably across many different branches that are updated at varying speeds, currently running Ubuntu 16.04, so effectively stretch)