Esy integration

[bsansouci]

I'll start with a small explanation and we'll go from there.

Esy's a package manager for Reason. It allows using any build system to build packages, as long as the build system obeys a bunch of rules. Right now bsb-native (and bsb) do not obey the rules to work under Esy. There are two big chunks of work that we can identify:

• bsb-native needs to output findlib (ocamlfind) files (a file called META and packagename.install see http://projects.camlcity.org/projects/findlib.html) so that Esy can read them. This needs to be a command line argument that will build using ocamlfind (always) and generate the files.
• bsb-native needs to be relocable (this will accelerate build install / build times because Esy will cache the binaries generated the first time bsb-native's built and any package depending on bsb-native will simply reuse the artifacts)

The first bullet I'm almost done with. I'm pretty simply merging opam_of_pakagejson into bsb-native and allowing the user to generate the findlib files. I've been having some trouble really testing that the files are correct though, still needs a bit of love but I'll push to a branch and tag this issue.

The 2nd bullet point's a bit of a pickle but is not strictly needed for cooperating with Esy. The problem is that bsb makes assumptions of where the stdlib is relative to where bsb.exe is (same for where ocaml is etc...). Those assumptions do not hold in Esy world when we enable global caching, unfortunately.

Those are the things that we can do inside of bsb-native. That said, this does not allow any Esy user to access any bsb-native package. One would have to go to all bsb-native packages and update their package.json to contain new Esy-specific fields like such. I think we could very easily make Esy understand how to call bsb-native (simply detecting the presence of a bsconfig.json file, or a devDependency on bsansouci/bsb-native). All that Esy needs to do is call bsb -make-library (literally), because Esy builds in order and we'd be building using ocamlfind which will have its environment all setup by Esy automatically. The reasons for making Esy understand bsb is two fold:

• it'll allow us to support all bsb (and bsb-native) packages without having to update each one of them
• it'll allow for less friction when setting up a new Reason native project. Each time you'd have to add the build command which would always be the same one.

Would love some thoughts and help with any of this <3

[jordwalke]

I'm actually working on plain JS bsb right now and it's looking like this is not going to be hard at all to support. (I haven't looked too deeply into the bsb-native stuff, except for the amount we experimented with together last week). Happy to answer any other questions.

[jordwalke]

bsb-native needs to be relocable (this will accelerate build install / build times because Esy will cache the binaries generated the first time bsb-native's built and any package depending on bsb-native will simply reuse the artifacts)

Someone might interpret this as meaning that the binary bsb-native (the build system executable) needs to be relocatable. That would be an incorrect interpretation. bsb-native itself doesn't actually need to be relocatable - esy will force it and any other binary to be relocatable.

[jordwalke]

Each remaining issues is either an important blocker, or merely perf:

1. [PERF] bs-platform (and bsb-native) do not yet model a dependency on the ocaml compiler so that means that even if you've installed merlin, you have to recompile a small ocaml from scratch. (a concern to some corporate environments or arch-linux who require rebuilding). I made a release of bs-platform that does model its dependencies on the npm ocaml compiler. See how I did it here. We can even just drop an esy.json file next to package.json and esy will pay attention to esy.json while npm would pay attention to package.json.
2. [PERF] bsb-native (and bsb) do not allow placing build artifacts in _build. Either bsb-native can place its build artifacts into _build, or it can do even better than all the other build systems and place it directly into the build cache $cur__target_dir. We should solve this last, because this isn't a blocker - it's just going to make compilation less incremental.
3. [BLOCKER] bs-platform only works with npm and yarn, but not package managers like opam or esy that place build artifacts into a protected location to avoid corruption. NOTE: IF bsb-native could build opam packages then bsb-native would necessarily work work esy packages because esy can use any build system that works on opam. So what to do about it? (besides the ocamlfind integration which is going well).

Number three is the most important one to solve. Why does bsb-native a hard coded assumption about where the stdlib is? Isn't bsb-native just a binary that generates a ninja file which calls ocamlopt? I am working on solving number 3 for bs-platform JS compilation, but I didn't think it would be an issue for bsb-native since you just invoke ocamlopt.

[andreypopp]

I agree with the points @jordwalke listed.

Few more thoughts:

2. [PERF] bsb-native (and bsb) do not allow placing build artifacts in _build. Either bsb-native can place its build artifacts into _build, or it can do even better than all the other build systems and place it directly into the build cache. We should solve this last, because this isn't a blocker - it's just going to make compilation less incremental.

This is not a blocker for supporting bsb-native in general with Esy (building dependencies which use bsb-native once and put them to global store). But a real blocker for using bsb-native for development with Esy as without this it wouldn't be possible to use incremental compilation.

I want to stress though that it's more preferable to make bsb-native to support putting build artifacts into a location configured via $cur__target_dir environment variable as this is even more performant than what we have now with jbuilder (now dune) and its _build directory.