Idea: Executing code in memory and port this version to Windows

[kspalaiologos]

Hi, I've got an idea to execute code generated in JIT in memory (I'm going to release library to allow it) and port this to windows using asm sysenter's.

Or linking aganist LIBC / passing program function adress that code would call.

This would make things easier / faster. Porting program would give it a little bit more attention.

[skeeto]

The existing -e option already does execute JIT code in memory — skipping ELF generation altogether — provided the compiler is hosted on a supported platform. In theory, this tool can cross-compile, though it relies on system-specific headers (sys/syscall.h, elf.h). Take note of the "openbsd" branch.

The main challenge with porting this compiler to Windows is, unlike Linux, the lack of a stable system call interface. The stable interface is symbols found in Ntdll.dll or Kernel32.dll. To produce a PE32+ .exe you'd need to either generate an import table for either of these DLLs and route system calls through it, or write a linker to statically link against an available libc.

For in-memory JIT, which is what you said you intend, the easiest thing to do is change asmbuf_syscall() to generate an appropriate function call into libc (SYS_read, SYS_write, SYS_exit). This will be an already-known address (e.g. no linker needed), so that should be pretty simple, except that register allocation isn't prepared for it (rsi being clobbered by the function call).

How to do this portably in a way that fits into the existing architecture is another matter. Since I initially wrote this, I've realized that relying on a system-provided elf.h is a mistake (which I would never make again). Those constants and structures come from a specification and isn't system-specific, so they should just be hard-coded in this program the same way you'd hard-code pi. That eliminates a non-portable external dependency.

If sys/syscall.h is eliminated, replaced with hard-coded values for Linux, then this becomes a portable POSIX C program that can trivially cross-compile to Linux from any POSIX system. Make those system call numbers selected dynamically at run-time (option switch for selecting host OS), and you can cross-compile to other systems, like the already-supported OpenBSD, in the same build.

Make the in-memory execution feature optional (-e), then except for getopt() (trivially resolved), this becomes a portable, plain C program that can cross-compile to various unixes from anywhere, including Windows.

Finally, make asmbuf_syscall() a little more flexible so that it generates plain function calls on for certain platforms (e.g. Windows), as described previously, and you can JIT on those platforms. You'll also need replacements for POSIX mmap(), mprotect(), and munmap() — e.g. VirtualAlloc(), VirtualProtect(), VirtualFree() on Windows.