grain icon indicating copy to clipboard operation
grain copied to clipboard

feat(compiler)!: Custom Grain object files

Open ospencer opened this issue 2 months ago • 0 comments

Overview

This PR replaces Grain's intermediate .gr.wasm files with Grain object files. This is for a variety of reasons, including:

  • Debuggability. Our wasm-based linker is unable to reconstruct debugging/sourcemap information from wasm binaries. This approach once again generates proper sourcemaps. There is additional work to be done to make the sourcemaps truly usable, but this is a major step in the right direction.
  • Optimizations. This change will allow us to do "whole program" optimizations to better assist Binaryen with its optimizations.
  • Performance. Going from our IR to Binaryen IR and serializing is the bulk of compilation time. Since we no longer are serializing/deserializing many files, there is a performance boost. For a hello world program, cold compile time is down 10% and warm compile time is down 25%. When running in JS, cold compile time is down 15% and warm compile time is down 21%. This also gets subsecond warm builds in JS on my machine.

Technical details

Grain object files use the .gro extension. As no wasm files are specific to Grain, the default output filename from the compiler uses the .wasm extension rather than .gr.wasm.

Object files follow this layout:

magic version len version mash_program offset cmi mash_program
0xF0 0x9F 0x8C 0xBE 5 0.6.3 1024 <cmi> <mash_program>

mash_program does include the module's cmi, but this format allows typechecking to unmarshal just the type information.

We use OCaml's Marshal module as our binary format (and for Whole Grain, we can use our Marshal module). js_of_ocaml can't marshal floats into a format that can safely be read back by native code. I wanted object files to be the same between native and js, so we store floats as int64s in the mashtree to guarantee consistent behavior.

Reviewing this PR

The code delta for this PR is fairly small, but snapshot tests are now sexprs of mashtrees. As such, we have all new snapshots. I'd recommend clicking the first commit to review the code diff, or using the GitHub UI to hide changes to snapshots.

Closes #1903

ospencer avatar May 04 '24 00:05 ospencer