Rework racket based runtime/compilation

[dolio]

This PR includes the changes that get us about as far as we can easily go for our racket evaluation/compilation at the moment. It includes the following aspects.

• I renamed the runner file/executable to ucr, so that it's more similar to ucm. It's not something that people should be calling directly, but runner is obviously not a very good name.
• I changed the 'compilation' part of ucr to just generate a file. The reason for this is that somehow a math library we are using for the primops causes create-embedding-executable to fail. The easiest solution to this problem is to divide the compilation process into two steps: first generate the file using all the unison infrastructure, then compile that file. At first I was thinking about copying the math operations into the primops, but I realized that dividing the programs up is a better solution.
• At the moment, compiling the generated file is just done via raco exe using a racket installation. The reason for this is that create-embedding-executable does not seem to work in a standalone executable. The API needs to find a copy of the racket executable to modify, and the procedure for that doesn't work in a standalone binary (confirmed with racket devs). Peeking at the code, it doesn't seem like this absolutely couldn't work, but making it work would require someone digging into the racket internals to make it happen. So for the time being, compiling requires a racket install.
• I've included a short transcript that will download base, @unison/internal, and generate the racket modules alongside the checked-in modules. This is a step necessary in building ucr or getting things set up for compilation, so we'll probably use it when we get this stuff building in CI.

Even for compilation, where we still depend on a racket install, we no longer actually need to make people download @unison/internal themselves. We can run the library generation as part of a build, and just distribute the generated libraries to people. The runtime should be completely self-contained, though. Once ucr is built, ucm just needs to shell out to it to execute run.native. A racket install is only necessary to build ucr.

I think one tweak I have left is a better error message for compile.native if raco doesn't exist. Beyond that, I just need a little advice on how to get this hooked into CI to get everything distributed properly. I might need to make the native runtime calls a bit more configurable with respect to the location of ucr and such, too, but that shouldn't be a big deal.

[aryairani]

• I renamed the runner file/executable to ucr, so that it's more similar to ucm. It's not something that people should be calling directly, but runner is obviously not a very good name.

I'm thinking we could use a more descriptive name and stick it in libexec if nobody needs to call it.

• I changed the 'compilation' part of ucr to just generate a file.

What did it do before?

• I've included a short transcript that will download base, @unison/internal, and generate the racket modules alongside the checked-in modules. This is a step necessary in building ucr or getting things set up for compilation, so we'll probably use it when we get this stuff building in CI.

Even for compilation, where we still depend on a racket install, we no longer actually need to make people download @unison/internal themselves. We can run the library generation as part of a build, and just distribute the generated libraries to people. The runtime should be completely self-contained, though. Once ucr is built, ucm just needs to shell out to it to execute run.native. A racket install is only necessary to build ucr.

Can we just do all of this in CI? Build ucr there, and nobody needs to install racket outside of CI?

I think one tweak I have left is a better error message for compile.native if raco doesn't exist. Beyond that, I just need a little advice on how to get this hooked into CI to get everything distributed properly.

We can pair on the CI part if you want.

[dolio]

I'm thinking we could use a more descriptive name and stick it in libexec if nobody needs to call it.

Any suggestion on the name? I guess it could be something more like unison-runtime or something.

What did it do before?

It generated a file then threw errors. But the command line gave you the expectation that it would produce an executable.

Can we just do all of this in CI? Build ucr there, and nobody needs to install racket outside of CI?

You can do the equivalent of run. Or you can generate a racket source file, but that's useless without racket. If you want to build a standalone executable from some unison code, you need racket installed, because the API for generating executables doesn't work when you build it into a standalone executable.

[ChrisPenner]

Looks great! I'd second the idea of changing ucr to something like unison-runtime

[aryairani]

When would racoErrMsg show up currently? Current plan is to never have to call it from within ucm, right? (It doesn't need to be deleted yet, but maybe once CI is working?)

[dolio]

racoErrMsg shows up when you run compile.native and racket isn't installed. You need racket to compile standalone executables right now.

[aryairani]

Heads up that I'm seeing some intermittent failures in the interpreter tests on macOS. https://github.com/unisonweb/unison/actions/runs/7937787444/job/21675781417?pr=4675#step:16:173

Any chance that's related to this PR? Otherwise I can hope for green and open a separate ticket.

[aryairani]

Note to self: Currently the interpreter tests are being run in the "build JIT binary" stage, which is a little wrong. But also maybe that's fine / easier because they both rely on the result of base.md and it might be a pain to factor that out

[dolio]

Okay, I think the tests in @aryairani/jit-tests should pass with this branch.

I also added compiler.generateSchemeBoot to @unison/internal in the new release 0.0.11 that this depends on. I think you were planning on doing something with that in the transcripts?

[aryairani]

Okay, I think the tests in @aryairani/jit-tests should pass with this branch.

I also added compiler.generateSchemeBoot to @unison/internal in the new release 0.0.11 that this depends on. I think you were planning on doing something with that in the transcripts?

Thanks for the reminder, got it in 20a0e73.

[aryairani]

~The default runtime location is set in fixNativeRuntimePath, currently as $XDG_DATA/unisonlanguage/libexec/unison-runtime, and can be overridden by nativeRuntimePathFlag.~

• [x] It should probably default to a system-wide location though, like $0/unison-runtime?

The default runtime location is set in fixNativeRuntimePath, as $0/runtime/unison-runtime[.exe], and can be overridden by nativeRuntimePathFlag.

[aryairani]

A racket install is only necessary to build ucr.

The racket install is still needed to use compile.native, right?