Add LLVM support for JIT linking with a static binary

[jeaye]

In order for jank to be statically compiled, or for it to AOT compile jank programs to static binaries, either of which use JIT compilation, we need to add support for LLVM's JIT linker to find symbols in static binaries.

By default, symbols are in the .dynsym section of an ELF. However, static binaries don't have this section. Symbols are instead in the .symtab section. This task will involve learning a fair amount about ELF and these symbols.

Note: Focus on Linux/ELF first. We can tackle macOS binaries after.

Steps

• Make a static jank build
  • Add -static to the AOT compilation flags
  • Verify binary is static with ldd
• Add a simple void jank_test_link(); fn which logs to stdout when called to main.cpp
• Use the jank cpp-repl to declare (not define) and then call the fn
  • Verify this works for you with a non-static build first, to ensure your test is sane
• Patch LLVM for static symbol support
• Re-run test until it works!

Patching LLVM

From the LLVM Discord, @lhames (an LLVM dev), said the following to me:

---

There are extension points for generators in both the C and C++ APIs. For the C++ API you'd write a custom DefinitionGenerator class and add it to a JITDylib (I'd use your main JITDylib to start with) using addGenerator.

What your describing would make sense as a generic LLVM utility, so I'd say write it as a new class in its own header in llvm/include/llvm/ExecutionEngine/Orc -- ELFStaticSymtabGenerator or something like that. The generator would take the address of an ELF header and then implement tryToGenerate (https://github.com/llvm/llvm-project/blob/8549b324bc1f450f4477f46f18db67439dbf6d75/llvm/include/llvm/ExecutionEngine/Orc/Core.h#L958) by using libObject to parse the symbol table, and absoluteSymbols to add the requested symbols into the JIT.

---

Support

If you need guidance, ask in the #jit channel on the LLVM Discord.

[stmonty]

I'll tackle this!

[elken]

Extremely minor progress on this but I have Boost compiled statically now.

I'll drop the other items I was going to look at here to make eg libzip static 🙏 This would be a great one to get in

[jeaye]

Extremely minor progress on this but I have Boost compiled statically now.

I'll drop the other items I was going to look at here to make eg libzip static 🙏 This would be a great one to get in

How are you getting boost/libzip/etc static? Is it something we can do in a cross-platform way? My understanding is that we'd have two options:

1. If the distro has the static lib packaged, install it
2. Build from source

From what I've seen, static libs being packaged is hit or miss, so I'm very interested in how you see us making this work.

[elken]

So Boost is easy, its just a CMake argument, and that's as far as I got with it

[stmonty]

Fmt can also be made static as we are vendoring it in and it can be built statically as it has no other dependencies. libzippp requires libzip being available, and it seems Ubuntu doesn't package libzip.a, just libzip.so for example.