Add boot command to helper library

[tobil4sk]

This is based on the nekotools boot -c feature from neko. It allows running:

haxelib run hashlink boot main.hl

Which generate main.c, which is just the hl entry point with the bytecode embedded. It is then possible to compile this file with a simple compilation command:

cl.exe main.c /MD /I C:\HaxeToolkit\hashlink\include /link /libpath:"C:\HaxeToolkit\hashlink" libhljit.lib libhl.lib user32.lib
#or
gcc -o main main.c -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib

This is a bit like hlboot.dat, however it has some advantages:

• No unwanted cli handling
• Executable and bytecode are in a single file
• No arbitrary bytecode file being loaded and executed from CWD

Compared with hlc:

• It has a much faster compilation time
• Runtime performance may be slower than hlc, as it still runs with jit.
• Unlike hlc this only supports x86, again due to the jit.

To make this work, I had to add a libhljit static library, which just contains everything in hl.exe except main.c.

[ncannasse]

Instead of requiring a libhljit installed locally, why not simply distributing the required files as part of the haxelib projet, then packing them together into a single .c file (with bytecode) that can be compiled with no other dependency than libhl ?

[tobil4sk]

Instead of requiring a libhljit installed locally, why not simply distributing the required files as part of the haxelib projet, then packing them together into a single .c file

A single main.c that includes everything would be quite a bit slower to compile. The current small c file with the precompiled static libhljit gives for a very quick compilation time.

I guess as a middle ground, the source files could be distributed with the haxelib and haxelib run hashlink boot could precompile the files on the first run into an internal static libhljit so it doesn't have to be compiled again. The downside is that the tool has to know how to compile all the other files, which is more involved than compiling a simple main.c, so some makefiles/vcxproj files would have to be distributed too.

This would mean we don't have to depend on the stability of libhljit, which is a benefit as libhl is probably more likely to be stable (since hlc also depends on its stability). This would make it less of a problem if the hashlink haxelib is out of sync with the libhl installed on the system.

[ncannasse]

A single main.c that includes everything would be quite a bit slower to compile.

I don't think so, C is very fast to compile and there's only few files for the JIT. Unless you're looking at something like 100ms optimization ? Give it a try, I'm sure it will be almost unnoticeable.

[tobil4sk]

Give it a try, I'm sure it will be almost unnoticeable.

I'm compiling haxelib for testing:

# Small run.c with libhljit
$ time gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib
gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib  0.22s user 0.03s system 99% cpu 0.251 total

# Merged run.c
$ time gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib
gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib  3.43s user 0.04s system 98% cpu 3.511 total

[ncannasse]

A bit more than I was expecting, I guess O3 has big cost. Given it's nice to have a single C code for distribution I think it's still worth the 3 extra seconds.

Le sam. 18 janv. 2025, 13:15, tobil4sk @.***> a écrit :

Give it a try, I'm sure it will be almost unnoticeable.

I'm compiling haxelib for testing:

Small run.c with libhljit

$ time gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib 0.22s user 0.03s system 99% cpu 0.251 total

Merged run.c

$ time gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib 3.43s user 0.04s system 98% cpu 3.511 total

— Reply to this email directly, view it on GitHub https://github.com/HaxeFoundation/hashlink/pull/744#issuecomment-2599693378, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZXQAMVRFRDWW5BMQSYND2LJAWJAVCNFSM6AAAAABVMHSEZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJZGY4TGMZXHA . You are receiving this because you commented.Message ID: @.***>

[Simn]

I'd actually be interested in having both options.

[ncannasse]

One nice thing could also be to compile the exe entirely statically as we do on consoles so we have a single binary with no libhl dependency

Le sam. 18 janv. 2025, 15:09, Simon Krajewski @.***> a écrit :

I'd actually be interested in having both options.

— Reply to this email directly, view it on GitHub https://github.com/HaxeFoundation/hashlink/pull/744#issuecomment-2599731351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZXQELZYYYKXVZM2ZNLJ32LJOBZAVCNFSM6AAAAABVMHSEZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJZG4ZTCMZVGE . You are receiving this because you commented.Message ID: @.***>

[tobil4sk]

Given it's nice to have a single C code for distribution I think it's still worth the 3 extra seconds.

The goal is to replace nekotools boot -c for haxelib compilation with a hashlink solution. Having this delay is a noticeable downgrade from how it is now with neko.

I think it's a valid concern to not want libhljit to be distributed with the hashlink binaries, since there is no guarantee of stability. It makes more sense to have the sources distributed with the hashlink haxelib, then the only concern is whether the hashlink haxelib is compatible with the libhl installed, which is a simpler requirement (that is shared by hlc).

Ultimately, the details of main.c are not really important here, it is an implementation detail that the user shouldn't have to worry about. If automatic compilation is added to the boot command, we could hide all those concerns and just take a main.hl and output main.exe.

haxelib run hashlink boot run.hl

Then the tool is free to decide how to deal with the files, whether to combine them into one or to do it separately with intermediate object files.

How about the following changes to the PR:

• libhljit is removed from the installation
• the required sources for compiling a stripped hashlink jit are distributed with the hashlink haxelib
• git versions will just use the hashlink sources directly (can also support HASHLINK environment variable)
• hashlink boot compiles immediately instead of giving you a c file to compile manually

[tobil4sk]

One nice thing could also be to compile the exe entirely statically as we do on consoles so we have a single binary with no libhl dependency

I know it is possible to compile everything statically for hlc, but is this possible at all with jit? I tried it but it failed when trying to load std functions, so I assumed that method only works for hlc. If it could work, that would also be useful for quickly making a haxelib binary with no libhl dependency (assuming static libhl and all hdlls are pre-compiled).

[ncannasse]

One nice thing could also be to compile the exe entirely statically as we do on consoles so we have a single binary with no libhl dependency

I know it is possible to compile everything statically for hlc, but is this possible at all with jit? I tried it but it failed when trying to load std functions, so I assumed that method only works for hlc. If it could work, that would also be useful for quickly making a haxelib binary with no libhl dependency (assuming static libhl and all hdlls are pre-compiled).

We would need to try to load in the current binary before trying to load the hdll, it should be safe to do so.