Native or wasm compiler backend for Gleam

[lpil]

It would be fun and also useful if we could compile Gleam to native code. This comes up frequently in our discord chats, so let's keep some notes here for future reference.

References

Codegen

https://leonardschuetz.ch/blog/nan-boxing/

https://piotrduperas.com/posts/nan-boxing (with benchmark)

https://bernsteinbear.com/blog/compiling-a-lisp-2/

WASM

https://github.com/WebAssembly/binaryen

https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html (used by TinyGo)

Memory layout

Wren

https://wren.io/performance.html

OCaml

See section 2, the value type https://v2.ocaml.org/manual/intfc.html

https://dev.realworldocaml.org/runtime-memory-layout.html

https://www.janestreet.com/tech-talks/unboxed-types-for-ocaml/

https://lor.sh/@[email protected]/109473362564740302 (https://gist.github.com/Armael/aef1b9cec5d4d9591ef19dc6679faa5d)

Grain

https://github.com/grain-lang/grain/blob/main/docs/contributor/data_representations.md

Scheme

https://github.com/racket/ChezScheme/blob/8d134111b5a12ff9d0fecfcdb65e1a7ca64ad5e2/IMPLEMENTATION.md

https://www.youtube.com/watch?v=BcC3KScZ-yA

Skybison

https://github.com/tekknolagi/skybison/blob/trunk/doc/object-model.md

https://github.com/tekknolagi/skybison/blob/trunk/runtime/objects.h

[rockwotj]

Any interest in directly compiling to WASM?

[lpil]

edit: I have changed my mind! wasm is good target.

[dch]

Perhaps it’s possible to have an erts single image binary and slap a gzip of beam files on the end of it?

[rockwotj]

Warp has an Erlang example: https://github.com/dgiagio/warp/blob/master/examples/erlang/README.md

[lpil]

Perhaps it’s possible to have an erts single image binary and slap a gzip of beam files on the end of it?

@dch I would like that but in the far future I'd also like a native Gleam that could be used when more single threaded performance or minimal memory usage is required. It could integrate with Erlang based Gleam using NIFs. One for the far future :)

[dch]

@rockwotj that's neat but basically a self-extracting archive with some smart caching. I'd like to have erts itself be able to handle this case. Not really a topic for here, though!

[dch]

@lpil oh yeah I see what you meant now. I'd love fast-executing command line tools, for example. No need for NIFs initially.

[rockwotj]

For what it's worth I've been (slowly) hacking on a C++ target output for Gleam.

I chose C++ over C because templates, (and lamdbas) are really nice to enforce typesafety and not mess with a bunch of void pointers that plain C would need.

So far math expressions, functions, records (with hierarchy and shared field support) and reference counting via std::shared_ptr are supported. There is certainly a lot of work to go, but I hope to slowly chip away at progress. There are a couple of open questions around things like external types and how the higher level features above the package-compiler will work with complexity of invoking C++ compilers (let alone supporting all the platform specific stuff and things like dependency management).

Initially I really wanted to output golang, as it has a lot of nice parallels with erlang (goroutines, GC, error handling is "result based", etc), but Go has a very different module system than Gleam and generics are lacking in some aspects.

Anyways I am happy to discuss further and I'll make sure to post if I ever "catch up" with all the features in the language to discuss upstreaming some of this work if there is interest.

[lpil]

That's very cool! C++ would probably be what I go for if I were to make a native backend today. Looking forward to seeing what you make :)

I feel there likely is a shared IR that both JS and C++ could use. A simple proceedural language which typed Gleam could be lowered to. Possibly not practical right now, but perhaps in future.

[inoas]

A shared IR would mean we could write arbitrary targets against for that IR? Say one on JVM/Java?

Edit: If FFI could also target that IR language, could we then abstract away FFI for common libs so that they run on all targets supporting the IR?

[lpil]

A shared IR would mean we could write arbitrary targets against for that IR? Say one on JVM/Java?

It would reduce the amount of work to add or maintain procedural language backends, aye.

If FFI could also target that IR language, could we then abstract away FFI for common libs so that they run on all targets supporting the IR?

I think you'd still need code per platform as they'd all have different APIs and characteristics

[rockwotj]

Firefly might be interesting for gleam. You could support WASM by leveraging the existing Erlang target and not have to change semantics. If you wanted to be able to then run them without a wasmengine it seems like wasmer can do that.

[lpil]

Yup! We've spoke in the past about Gleam exposing type information to Firefly so we can enable LLVM type based optimisations too.