dune
dune copied to clipboard
ocaml
Dune should run itself when a package build command tries to run `dune`
If the dune executable currently running isn't what dune resolves to in PATH, packages that invoke dune in their build command should still start a copy of the current dune rather than the one from PATH. This situation arises when running a development version of dune where the exe is named main.exe.
note that this should extend to (run sh -c dune) as well. In other words, we should set PATH so that the current dune is included.
Good point; it's not sufficient to detect when the first word of the command is dune and replace it with the value of argv[0]. Setting PATH is also insufficient as it wouldn't handle the case where your dune binary is named something else (e.g. if you build dune yourself and it's still in _build/default/bin/main.exe). What about creating a temporary directory somewhere and copying/linking the current dune exe there (named "dune") and then putting that directory at the start of PATH? Is that too unix-specific for it to work on windows?
@gridbugs - that trick is entirely fine for Windows and indeed opam’s prototype compiler packaging is using a very similar trick to shim GCC out of Cygwin
You can get the current executable also on Windows like this (from @dra27's relocatable compiler branch), however unlike Unix.readlink on /proc/self/exe probably not in pure OCaml.
I'm not sure to understand what is the advantage of using dune recursively? How does this work if the constraint for dune doesn't match the current one? Isn't that also contrary to the idea of a lock file?
@kit-ty-kate good point. And the lockdir does contain a lockfile for dune when a project's dependencies depend on dune, so we should be building dune as part of the project's build and making the resulting exe available to all packages that depend on it. I wonder that's not currently happening in my experiments. I'll investigate further.
What about creating a temporary directory somewhere and copying/linking the current dune exe there (named "dune") and then putting that directory at the start of PATH? Is that too unix-specific for it to work on windows?
This is likely the best.
so we should be building dune as part of the project's build and making the resulting exe available to all packages that depend on it
That can't work because the dune that will be build will read/write metadata in a different format.
That can't work because the dune that will be build will read/write metadata in a different format.
Which metadata files are you referring to? If it's just metadata files within the package it's building then what would be the problem?
The dune-package fill produced when building a package must be compatible. This is what dune reads to make libraries, package sites, and other metadata available.
But why does the dune that's orchestrating the build need to read the dune-package files of individual packages from the project's dependencies? Dependencies don't need to build with dune at all so I don't see why dune can't treat the dune package as a foreign build system that happens to be called "dune".
Which dune do you plan to use to build the packages that are inside the user's workspace? E.g.
$ dune build bin/foo.exe
Will that dune used to install dune, or the dune that the user already had?
For packages in the current workspace I think it makes sense to use the dune the user already has (ie. the one orchestrating the build). But for packages in the lockdir that build with dune, I'm considering treating their dune dependency the same way we could treat any other build dependency of a package, building the dune package of the appropriate version and making its binary available to build the package.
For packages in the current workspace I think it makes sense to use the dune the user already has
Okay, so this dune will not be able to read the metadata generated by the inner dune. To discover the metadata about the packages we just built, we need to read the dune-package files.
Why is it important that the outer dune be able to read metadata generated by the inner dune? Not all packages build with dune and the outer dune can still handle the ones that don't.
For packages that are built without dune, we rely on findlib to provide partial metadata. It's important to note that this metadata is limited, and dune packages can provide much richer metadata which we do indeed use to provide many advanced dune features.
Wait did you mean to say that packages that aren't built with dune rely on findlib to provide partial metadata? I imagine we wouldn't need findlib when working with packages built with the same version of dune.
Makes sense. Can you elaborate on the advanced dune features that will take advantage of cases where non-workspace dependencies build with dune (or point to documentation). Also it sounds like we should constrain dependencies to those compatible with the current version of dune. Is that right?
Can you elaborate on the advanced dune features that will take advantage of cases where non-workspace dependencies build with dune (or point to documentation)
Here's a partial list: ppx rewriters, ppx drivers, virtual libraries, inline test drivers, privately installed libraries, custom installation sites. Essentially, anytime we need to implement some advanced features, we need to tweak this metadata to make it work for installed dune packages. This metadata has no compatibility guarantees and we are able to modify it whenever we want.
Also it sounds like we should constrain dependencies to those compatible with the current version of dune. Is that right?
We already do that. Sounds like we should drop dune from the lock directory.
By the way, our end goal is not to launch any external dune processes to build packages. This is going to give us some rather significant performance improvements.