Relocatable ocamlfind

[Leonidas-from-XIV]

This PR mirrors the work on the compiler on making it relocatable. The way it works it adds a new option for configure that can be used to instead of hardcoding the absolute path, it will resolve $PREFIX to the installation location using various means (readlink, reading environment variables etc).

[gridbugs]

I did some playing around with this branch while investigating https://github.com/ocaml/dune/issues/8931. I'm trying to build https://github.com/ocaml/Zarith directly (without dune) after installing ocamlfind from this branch without passing the new flag to ./configure and I get a suspicious error running ocamlfind install (from zarith's make install):

ocamlfind: Config file not found - neither @CONFIGFILE nor the directory @CONFIGFILE.d

That's suspicious to me because I thought that @CONFIGFILE was a placeholder for a value chosen at ocamlfind's compile time so it shouldn't make it into the binary.

My experimental setup is to clone this branch and run:

./configure \
    -bindir  /tmp/ocamlfind/bin \
    -mandir  /tmp/ocamlfind/man \
    -mandir  /tmp/ocamlfind/man \
    -sitelib /tmp/ocamlfind/lib \
    -config /tmp/ocamlfind/lib/findlib.conf \
    ;

make all
make install

export PATH=/tmp/ocamlfind/bin:PATH

...and then to run ./configure && make && make install from a checkout of zarith.

I haven't tried out the new configure option yet but just wanted to raise this as it looks like a regression. The above experiment works for me when I repeat it with the tip of the master branch of ocamlfind.

[Leonidas-from-XIV]

@gridbugs That's a good point, I missed the final @ of @CONFIGFILE@ that indeed breaks the code. I've pushed a fix.

[gridbugs]

Trying this out on macos and it looks like it's trying to read the (non-existent) /proc/self/exe:

$ dune build
...
ocamlc -c oneshot_webserver.mli
ocamlfind ocamlopt -package unix -c oneshot_webserver.ml
Uncaught exception: Unix.Unix_error(Unix.ENOENT, "readlink", "/proc/self/exe")
make: *** [oneshot_webserver.cmx] Error 3
-> required by _build/_private/default/.pkg/oneshot-webserver/target

(this was while trying to build https://github.com/gridbugs/oneshot-webserver-app/)

[Leonidas-from-XIV]

Indeed, I probably need to copy more of how the relocatable compiler deals with determining the executable name. Looks like on NextStep I can use _NSGetExecutablePath to determine this, however in the context of ocamlfind it needs I need to add an extra stub for it.

Maybe that's actually better, then I can replace Unix.readlink with a C stub, so I can probably avoid the dependency on Unix.

[gridbugs]

Since your latest changes I can now use this on macos and linux to build packages with dune's package management features. One issue I run into now is that using the ./configure arguments from ocamlfind's opam file now gives this error when linking ocamlfind:

ocamlc -I +compiler-libs  -o ocamlfind -g findlib.cma unix.cma \
           -I +unix -I +dynlink ocaml_args.cmo frontend.cmo
make[1]: Leaving directory '/home/s/src/oneshot-webserver-app/_build/.sandbox/ba495fce988e4e0fef6e323f48198258/_private/default/.pkg/ocamlfind/source/src/findlib'
File "_none_", line 1:
Error: Error while linking findlib.cma(Fl_bindings):
       The external function `fl_executable_path' is not available
make[1]: *** [Makefile:55: ocamlfind] Error 2
make: *** [Makefile:14: all] Error 2
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie

The problem goes away if you pass -custom to ocamlc which can be achieved by not passing -no-custom to ocamlfind's configure script.

[gridbugs]

I tried using this to build the topkg package and ran into some issues which I describe here: https://github.com/ocaml/dune/issues/10271.

In summary:

• the generated "topfind" file currently contains paths inside the temporary build sandbox which has been deleted by the time that topkg's build script attempts to open "topfind" with the #use directive
• the bytecode versions of the findlib libraries can't be loaded into the bytecode interpreter due to the inclusion of c stubs. The "topfind" file will attempt to load "findlib.cma" into the bytecode interpreter while building topkg which no longer works (I confirmed that removing the stubs fixes this).

More details in the linked issue.

[Leonidas-from-XIV]

I've removed the C parts, since I don't think there's a way to determine the location from OCaml on platforms other than Linux without at least some C code (e.g. both macOS and Windows).

Instead I made it look up OPAM_SWITCH_PREFIX first, which is set by OPAM and could be set by Dune when it builds non-Dune packages.

The topfind problem is indeed an issue, as I am not sure #use takes a variable and without that we can't compute the right path. Maybe this could be worked around by replacing topfind with a wrapper that generates a topfind in a temporary location and then loads that instead.

[gerdstolpmann]

ok, this seems to be the best PR so far to add some relocatibility. As far as I understand, the PR needs a special version of OCaml to work properly, right? Can we test for this in the configure script, and only allow relocatable paths when this version is available?

Regarding the #use topfind problem. Instead of #directory you can also call Topdirs.directory as a regular function. So there is a way to compute the install location. You could copy the code from findlib_config.ml into the topfind script when it is generated.

[Leonidas-from-XIV]

For the relocation it shouldn't need any specific compiler, the compiler patches are for relocating the compiler itself. I'm developing this on a normal OCaml 3.08.4 switch. Originally I thought I could use the same approach as the relocatable compiler patches do, but thanks to @gridbugs testing it turned out that I had to scrap the idea and the environment variables set by OPAM and Dune are probably a better option.

Excellent hint wrt. Topdirs.dir_directory! This has allowed me to port the code for the detection to topfind. It's a bit ugly as there is much duplicated code and it uses cat to stitch it together for the rd0 and rd1 variants.

I thought about unifying rd0 and rd1 which would simplify the preprocessing, since the only difference seemed remove_directory (could be done in a similar way as PPXOPT_BEGIN/END is implemented), but it seems that rd0 also doesn't handle cmxs so I am unsure whether this is a deliberate choice or just accidental divergence.

[Leonidas-from-XIV]

Hi @gerdstolpmann, I think this is ready as far as we are concerned. @gridbugs and I tested it on macOS and Linux as well as in Dune. I've just force-pushed to skip over all the intermediate steps in finding a proper solution and merging the commits fixing issues that were discovered on the way.

I took some extra care that #use "topfind" wouldn't trigger additional - : unit = () messages and now both variants with and without directories are generated from the same file to improve maintainability.

Could you take a look again and tell me if there's anything missing?