bsb-native
bsb-native copied to clipboard
Esy integration
I'll start with a small explanation and we'll go from there.
Esy's a package manager for Reason. It allows using any build system to build packages, as long as the build system obeys a bunch of rules. Right now bsb-native (and bsb) do not obey the rules to work under Esy. There are two big chunks of work that we can identify:
- bsb-native needs to output findlib (ocamlfind) files (a file called
META
andpackagename.install
see http://projects.camlcity.org/projects/findlib.html) so that Esy can read them. This needs to be a command line argument that will build using ocamlfind (always) and generate the files. - bsb-native needs to be relocable (this will accelerate build install / build times because Esy will cache the binaries generated the first time bsb-native's built and any package depending on bsb-native will simply reuse the artifacts)
The first bullet I'm almost done with. I'm pretty simply merging opam_of_pakagejson into bsb-native and allowing the user to generate the findlib files. I've been having some trouble really testing that the files are correct though, still needs a bit of love but I'll push to a branch and tag this issue.
The 2nd bullet point's a bit of a pickle but is not strictly needed for cooperating with Esy. The problem is that bsb makes assumptions of where the stdlib is relative to where bsb.exe
is (same for where ocaml is etc...). Those assumptions do not hold in Esy world when we enable global caching, unfortunately.
Those are the things that we can do inside of bsb-native. That said, this does not allow any Esy user to access any bsb-native package. One would have to go to all bsb-native packages and update their package.json to contain new Esy-specific fields like such. I think we could very easily make Esy understand how to call bsb-native (simply detecting the presence of a bsconfig.json
file, or a devDependency on bsansouci/bsb-native
). All that Esy needs to do is call bsb -make-library
(literally), because Esy builds in order and we'd be building using ocamlfind which will have its environment all setup by Esy automatically.
The reasons for making Esy understand bsb is two fold:
- it'll allow us to support all bsb (and bsb-native) packages without having to update each one of them
- it'll allow for less friction when setting up a new Reason native project. Each time you'd have to add the build command which would always be the same one.
Would love some thoughts and help with any of this <3
I'm actually working on plain JS bsb right now and it's looking like this is not going to be hard at all to support. (I haven't looked too deeply into the bsb-native stuff, except for the amount we experimented with together last week). Happy to answer any other questions.
bsb-native needs to be relocable (this will accelerate build install / build times because Esy will cache the binaries generated the first time bsb-native's built and any package depending on bsb-native will simply reuse the artifacts)
Someone might interpret this as meaning that the binary bsb-native
(the build system executable) needs to be relocatable. That would be an incorrect interpretation. bsb-native
itself doesn't actually need to be relocatable - esy
will force it and any other binary to be relocatable.
Each remaining issues is either an important blocker, or merely perf:
-
[PERF] bs-platform (and bsb-native) do not yet model a dependency on the ocaml compiler so that means that even if you've installed merlin, you have to recompile a small ocaml from scratch. (a concern to some corporate environments or arch-linux who require rebuilding). I made a release of bs-platform that does model its dependencies on the npm ocaml compiler. See how I did it here. We can even just drop an
esy.json
file next topackage.json
andesy
will pay attention toesy.json
whilenpm
would pay attention topackage.json
. -
[PERF]
bsb-native
(andbsb
) do not allow placing build artifacts in_build
. Eitherbsb-native
can place its build artifacts into_build
, or it can do even better than all the other build systems and place it directly into the build cache$cur__target_dir
. We should solve this last, because this isn't a blocker - it's just going to make compilation less incremental. -
[BLOCKER]
bs-platform
only works withnpm
andyarn
, but not package managers likeopam
oresy
that place build artifacts into a protected location to avoid corruption. NOTE: IFbsb-native
could buildopam
packages thenbsb-native
would necessarily work workesy
packages becauseesy
can use any build system that works onopam
. So what to do about it? (besides the ocamlfind integration which is going well).
Number three is the most important one to solve. Why does bsb-native a hard coded assumption about where the stdlib is? Isn't bsb-native just a binary that generates a ninja file which calls ocamlopt
? I am working on solving number 3 for bs-platform
JS compilation, but I didn't think it would be an issue for bsb-native
since you just invoke ocamlopt
.
I agree with the points @jordwalke listed.
Few more thoughts:
- [PERF] bsb-native (and bsb) do not allow placing build artifacts in _build. Either bsb-native can place its build artifacts into _build, or it can do even better than all the other build systems and place it directly into the build cache. We should solve this last, because this isn't a blocker - it's just going to make compilation less incremental.
This is not a blocker for supporting bsb-native in general with Esy (building dependencies which use bsb-native once and put them to global store). But a real blocker for using bsb-native for development with Esy as without this it wouldn't be possible to use incremental compilation.
I want to stress though that it's more preferable to make bsb-native
to support putting build artifacts into a location configured via $cur__target_dir
environment variable as this is even more performant than what we have now with jbuilder
(now dune
) and its _build
directory.