ARM support

[erlanger]

Thanks so much for your prompt help. What is needed to have ARM support? (e.g. raspberry pi, android)

[pfalcon]

Supposedly, nothing ;-). That's the beauty of this approach - it should be as platform and system-independent as possible. Subject to actually proving that ;-). I stopped short of that - I have a lot of ideas and much less time to try them in their fullness.

So, just try to build the sample and run it and see how it goes.

[pfalcon]

(You of course won't be able to build with make STDLIB=0, as that uses x86_64 syscalls and instructions to call them. Would need a toolchain for a particular arch to build foreign_dlopen_demo (as make STDLIB=1, but perhaps with -static added) and toolchain for a particular target system for fdlhelper).

[erlanger]

I was trying it, still needs z_trampo.S for arm

[pfalcon]

Ah, yep, so there're still bootstrap arch-specific bits. But, there's nothing magic there. just for x86/x86_64: https://elixir.bootlin.com/linux/latest/source/arch/arm/include/asm/elf.h#L124 . So, you just set up SP, put zero somewhere (well, 3rd arg to "trampo" call, and jump to entry point).

[pfalcon]

$ cp micropython fdlhelper                                                                                                                      
woods_f:/data/local/tmp $ ./foreign_dlopen_demo                                                                                                                             
fdlhelper: can't open file '10e14': [Errno 2] No such file or directory

At least, that means following worked:

	.text
	.align	4
	.globl	z_trampo
	.type	z_trampo,%function
z_trampo:
	mov	sp, r1
	mov	r1, r0
	mov	r0, r2
	bx	r1

(Don't have an android toolchain around, so used a random binary I had around as fdlhelper.)

[erlanger]

Great got it working on 32bit pi, have not tested aarch64

[pfalcon]

Thanks for confirming. I'll clean up and push this stuff to the repo as time permits.

aarch64 for sure will need its own version of z_trampo.S.

[erlanger]

Great, look forward to it.

[pfalcon]

Ok, I tested it on RasPi4 32-bit too, and now it should build and run there out of the box (make ARCH=arm).

@erlanger: Can you please test and confirm that?

[erlanger]

@pfalcon sorry it took me a whille, testes on rapi2 (32bit) works great! Thanks

[pfalcon]

Thanks for confirming.

aarch64 support is in my backlog.

[erlanger]

Thanks, look forward to it, so that it works on phones

[AXKuhta]

It looks like aarch64 support is easy enough:

diff --git a/src/Makefile b/src/Makefile
index 84e5700..82e3166 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -6,7 +6,7 @@ SMALL = 0
 DEBUG = 0

 ARCHS32 := i386 arm
-ARCHS64 := amd64
+ARCHS64 := amd64 aarch64
 ARCHS := $(ARCHS32) $(ARCHS64)

 CFLAGS_i386 = -m32
diff --git a/src/aarch64/z_trampo.S b/src/aarch64/z_trampo.S
new file mode 100644
index 0000000..43a435c
--- /dev/null
+++ b/src/aarch64/z_trampo.S
@@ -0,0 +1,9 @@
+       .text
+       .align  4
+       .globl  z_trampo
+       .type   z_trampo,%function
+z_trampo:
+       mov     sp, x1
+       mov     x1, x0
+       mov     x0, x2
+       br      x1

Based on arm target, assembly needed some editing: - Register names: r0 -> x0 - Branch instruction: bx -> br

make ARCH=aarch64 will produce a working ./foreign_dlopen_demo

[AXKuhta]

Making it work on Android appears to be more tricky.

I am using Ubuntu 21.04 running in Linux Deploy to compile foreign_dlopen_demo, Termux to compile fdlhelper.

I don't know if it's caused by bionic libc or something else, but the internal state will get trashed somewhere after calling z_trampo(). The native write() function will segfault past this point, I think as a result of altered TPIDR_EL0. You should poke around with gdb yourself to get a better idea.

But z_dlopen and z_dlsym will work:

diff --git a/src/foreign_dlopen.c b/src/foreign_dlopen.c
index 66ba659..15df365 100644
--- a/src/foreign_dlopen.c
+++ b/src/foreign_dlopen.c
@@ -12,7 +12,6 @@ char *(*z_dlerror)(void);

 void do_jump(void **p)
 {
-       z_printf("do_jump: %p\n", p);
        z_dlopen = p[0];
        z_dlsym = p[1];
        z_dlclose = p[2];
diff --git a/src/foreign_dlopen_demo.c b/src/foreign_dlopen_demo.c
index 2a0f736..05abc47 100644
--- a/src/foreign_dlopen_demo.c
+++ b/src/foreign_dlopen_demo.c
@@ -1,3 +1,4 @@
+#include <unistd.h>
 #include "z_utils.h"
 #include "z_syscalls.h"
 #include "foreign_dlopen.h"
@@ -8,20 +9,24 @@ int main(int argc, char *argv[])
 {
        (void)argc;

+       if (getuid() != 0) {
+               z_printf("Need to be run as root\n");
+               z_exit(-1);
+       }
+
+       (void) chroot("/proc/1/root");
+       (void) chdir("/");
+
        init_exec_elf(argv);

-       init_foreign_dlopen("fdlhelper");
+       init_foreign_dlopen("/data/data/com.termux/files/home/fdlhelper");
+
+       void *h = z_dlopen("libc.so", RTLD_NOW);

-       z_printf("Come back: dlopen=%p\n", z_dlopen);
-       void *h = z_dlopen("libc.so.6", RTLD_NOW);
-       z_printf("Handle of libc.so.6: %p\n", h);
        void *p = z_dlsym(h, "printf");
        int (*_printf)(const char *fmt, ...) = p;
-       z_printf("Next line is printed by the printf() from libc.so:\n\n");
-       _printf("Hello from the other side!\n");

-       void *h2 = z_dlopen("libz.so.1", RTLD_NOW);
-       z_printf("\nHandle of libz.so.1: %p\n", h2);
+       _printf("Hello from the other side!\n");

        z_exit(0);
 }

[pfalcon]

@AXKuhta

It looks like aarch64 support is easy enough:

Nice, thanks for looking into that! (I got sidetracked and distracted by other things and didn't fire off my raspi4 too often, so didn't have a chance to look into it further.) Would you be interested to submit a PR?

[pfalcon]

I don't know if it's caused by bionic libc or something else, but the internal state will get trashed somewhere after calling ...

Well, this project is definitely an experimental one, which needs more testing to see what limits it has. (Which I intended to do, but ... have too many projects around.) And I was warned by Musl maintainers that ~~there may be some issues~~ it won't work, in particular issues with TLS (thread-local storage) access was cited. Then, it works at least with simple cases, which I find better than finding out that "this doesn't work at all, and nobody bothers to try to make it work".

Thanks for sharing your experiences, much appreciated!

[AXKuhta]

@pfalcon Sent a PR with the aarch64 changes.

[AXKuhta]

Didn't think that would work, but saving and then restoring TPIDR_EL0 is enough to make initial glibc functions work again: https://github.com/AXKuhta/foreign-dlopen/commit/733fd6f9266ca7f1c52cfbb4737222a2ee7fc411

It seems TLS is exactly what this system register is responsible for. This article mentions that TPIDR_EL0 is used to store the thread pointer, citing glibc source code.