pkg: dune sandboxing and non-relocatable packages

[gridbugs]

Lately I've been running into issues working on dune package management where certain key low-level packages are incompatible with dune's sandboxing rules. Raising this issue so we can have a conversation about whether our current approach is the right approach and to share this problem with a wider audience in case there's an angle I've missed.

The problem

When building and installing with opam, variables are set in the context of the destination directory structure within the appropriate switch, and as far as I can tell, install commands are allowed (expected?) to create files directly within the switch's directory structure. This differs from how opam packages are built by dune, where packages are built and installed into a package-specific temporary sandbox directory, and then installed artifacts are copied from the sandbox directory to a directory structure shared by multiple packages (similar to an opam switch). Dune's approach can break assumptions made by opam packages that variables set during their build/installation refer to their eventual destination (e.g. ocamlbuild: https://github.com/ocaml/dune/issues/10290). Dune devs have been referring to packages whose assumptions are broken in this way as "non-relocatable".

Possible solutions

To address non-relocatable packages we need to do one of the following:

• patch and upstream changes to any non-relocatable packages. Fortunately non-relocatable packages are rare, though it seems to mostly affect low-level packages with many reverse deps (e.g. ocaml, ocamlfind, ocamlbuild). This is the approach that we are currently taking. Long term we need to make sure that these packages don't break in a way that only manifests when building within dune's sandbox, probably with something resembling ocaml-ci but which builds packages with dune rather than opam.
• supply alternative .opam files for non-relocatable packages rather than using their .opam file from opam-repository. At least some of these packages do provide mechanisms within their build scripts to be relocatable, or at least to be moved to a specific location after building, but these mechanisms aren't used in their published .opam file. This is compatible with dune's sandboxing semantics but we may need to pass different flags to ./configure scripts and makefiles. Long term we would need to keep the alternative opam files up to date as their packages get new releases.
• build non-relocatable packages directly inside the shared filesystem within _build rather than the sandbox. Long term we would need to maintain the list of packages that we allow to build without a sandbox.

[rgrinberg]

I think you have the right idea. We require a tighter set of constraints for packages to be buildable by dune. The upside of that is that we offer a drastically improved build experience. Now this might be a bad trade-off if a large chunk of OCaml packages were not relocatable. But that is absolutely not the case. I can count 3 important packages that aren't relocatable: OCaml, ocamlbuild, ocamlfind. All the rest I can do without. Once we get over the hump with these initial 3, we're unlikely to ever look back at this issue.

build non-relocatable packages directly inside the shared filesystem within _build rather than the sandbox. Long term we would need to maintain the list of packages that we allow to build without a sandbox.

This has one key downside: it makes it impossible to cache the non-reloctable packages and all of packages that depend on them transitively. I'd say that's pretty unacceptable.