[pkg] Build OxCaml with Dune package management

[maiste]

This issue will serve as a meta issue to describe the changes required to build OxCaml with Dune package management.

Currently, I have trying to build it using @gridbugs small repository, hello-oxcaml. The locking happens correctly, but the build encounters various errors:

1. Building the init-compiler requires it to be added to toolchains supported list. Otherwise, as it is not relocatable, it will suffer the same problem as the compiler.
2. The init-*.pkg use the %{prefix} variable, which is causing a problem with Dune Package Management because it points to the _build/.sanbox/... path. We extend the path with the dependencies instead of having a global state as opam would do. The semantic of %{prefix} is not really clear for me in this context. I'm not confident it has one.
3. If we change the path using %{pkg:init-compiler:share} in the init-*.pkg lock files, we are able to build it to a findable path. However, when doing the variable expansion in other *.pkg files (which updates the PATH accordingly), it is not using the path within toolchains but the one within the _build directory. I'm writing a test to reproduce this error.
4. In the final stage, when building the ocaml-variants package (after building init-* packages), it fails because of the problem discovered in #11514. Indeed, (run autoconf) generates a configure but it can't be executed.

I have been discussing with @rgrinberg on how to make Dune Package Management worked with OxCaml and more broadly on how to generalize it to support non-relocatable packages.

I'll update the issue to point to the various resources and changes.

[dra27]

The opam packaging has been designed to work well for opam - in particular to allow OxCaml to be updated without having to rebuild all the constituent parts.

The obvious Dune fit would be to building OxCaml in a separate build context (i.e. compiling the Dune stage of OxCaml in a build context which has the correct compiler). I'm wondering (= quite strongly recommending) not adding features to dune pkg just to support the init-* packages, but rather how about looking at refactoring the packages themselves so that OxCaml is installable as a single package in the first instance?

[dra27]

Just as a proof-of-concept for simplifying:

--- a/dune-workspace
+++ b/dune-workspace
@@ -2,7 +2,7 @@

 (repository
  (name jst)
- (url git+https://github.com/janestreet/opam-repository#with-extensions))
+ (url git+https://github.com/dra27/opam-repository#with-extensions-dp))

 (lock_dir
  (repositories upstream overlay jst))

and then:

$ dune --version
"Dune Developer Preview: build 2025-04-26, git revision
1a0d3646a365210cb42a4366cb3a026b4c8928e3"
$ dune pkg lock
$ time dune build
 Downloading ocaml-variants.5.2.0+flambda2
    Building ocaml-variants.5.2.0+flambda2

real    12m59.186s

The first build is taking an inordinately long time[^1], but the caching is definitely working after that:

$ git clean -dfX
$ time dune build
    Building ocaml-variants.5.2.0+flambda2

real    0m1.483s

[^1]: I haven't figured out what's going on, but my hunch is it's a failure to detect the appropriate parallelism in the "internal" Dune call in OxCaml's build: forcing a higher -j brought the first-time build down to the ~6minutes that an opam switch switch with the same package was taking.

[gridbugs]

+1 to modifying the package metadata rather than modifying dune. Thanks for the proof-of-concept @dra27!

[gridbugs]

I've updated my example project to use @dra27's fork of the JST repo which internalizes the "init-*" packages and changed the example program to use the new oxcaml syntax to demonstrate that it's being built with the oxcaml compiler.

[maiste]

@dra27 thanks for the proof-of-concept! I agree with your point. Note that some problems we encountered here are related to non-relocatable packages in general. This isn't exclusive to OxCaml. We need to fix these problems anyway for Dune Package Management.

As @gridbugs, I have tried the solution, and I'm able to make it build and run the code!

[maiste]

Some feedback about it, I have been able to push a working version using @dra27 patches and using a custom repository. It can be found at https://github.com/gridbugs/hello-oxcaml/tree/with-custom-patch.

[shonfeder]

I've also reproduced the example here. IMO, we can close this issue with some documentation explaining how to set up a project to build with oxcaml. (But there is still a lot of related work to make working with oxcaml via dune pkg easier!

[shonfeder]

@art-w is planning to do the finishing (documentation) work here and discovery for followups.

[Alizter]

I've tested this and it works. Workspace:

(lang dune 3.20)

(lock_dir
 (repositories :standard oxcaml))

(repository
 (name oxcaml)
 (url "git+https://github.com/oxcaml/opam-repository.git"))

and project:

(package
 (name oxcaml_experiements)
 (depends (ocaml-variants (= 5.2.0+ox))))

[Alizter]

Autoconf was required, which wasn't the case for ocaml. That might be worth trying to improve.

[rgrinberg]

Shouldn't there be a conf-autoconf dependency in the opam file then?

[Alizter]

@rgrinberg That appears to be the case https://github.com/oxcaml/opam-repository/blob/a1ea0d33dd5662b89183f751c3fec566d7860b75/packages/ocaml-variants/ocaml-variants.5.2.0%2Box/opam#L11.

My point was not that it wasn't clear, but that it was surprising since it wasn't exactly a drop-in replacement for ocaml.

[rgrinberg]

I still don't really get the point then. Regular OCaml also depends on autoconf, so in what way it isn't drop-in?

[Alizter]

Regular OCaml uses autoconf to generate their configure scripts, but they don't use it when building:

• https://github.com/ocaml/opam-repository/blob/master/packages/ocaml/ocaml.5.4.1/opam
• https://github.com/ocaml/opam-repository/blob/master/packages/ocaml-compiler/ocaml-compiler.5.4/opam
• https://github.com/ocaml/opam-repository/blob/master/packages/ocaml-variants/ocaml-variants.5.4.0%2Boptions/opam
• https://github.com/ocaml/opam-repository/blob/master/packages/ocaml-option-flambda/ocaml-option-flambda.1/opam

OxCaml on the otherhand appears to be running autoconf on the fly.

End-result for the user is that you need the autoconf binary to build OxCaml but no for OCaml.

[dra27]

I still don't really get the point then. Regular OCaml also depends on autoconf, so in what way it isn't drop-in?

Regular OCaml doesn’t ever depend on autoconf… it’d be better if the OxCaml release tarballs included configure (which is how autotools is intended to be deployed - end users are not supposed to have to run it). Upstream simplifies this by choosing to commit configure (which is a commit crime, as it’s a generated artefact!)

[dra27]

It’s a material concern as the number of distributions building OxCaml grows, because autoconf compatibility is not guaranteed between versions (for example, 2.71 can’t build the branches of OCaml using 2.69 correctly)