Relocation issues [ocamlfind]

[AltGr]

trafficstars

I started testing with binary caching locally. To see how, use latest opam and see https://github.com/ocaml/opam/blob/master/shell/opam-bin-cache.sh (doc at the end).

Of course, many packages aren't relocatable yet, and the above setup allows to check the blockers. My goal here is to improve the situation step by step.

My first experiments didn't hit issues with the bytecode shebangs¹, but with ocamlfind. Hopefully they shouldn't be too difficult to solve:

• lib/findlib.conf refers to absolute directories. Allowing to make those relative to the conf file would be a solution
• the settings (destdir and path) can be overriden using OCAMLPATH and OCAMLFIND_DESTDIR, which could be another solution (they can be set from the opam file). However, OCAMLPATH can prepend but not override.
• setting OCAMLFIND_CONF=/dev/null may work to force an override, though, but is a bit blunt.
• the directory is also hardcoded in lib/ocaml/topfind and lib/toplevel/topfind. This one can't be overriden, and it turns out it's more problematic (breaks packages using ocaml setup.ml)
• even after working around all the above, libraries shipped with OCaml are still wrong: it seems setting OCAMLLIB is required even if ocamlc -where has it right.
• there is also lib/findlib/Makefile.config. Are there packages around that use that ?

¹ using the wrong bytecode interpreter is much less likely to happen and cause problems than linking the wrong libs or trying to install to the wrong place (this setup is combined with my namespacing hooks to ensure the latter doesn't happen)

[AltGr]

Here is what I added to ocamlfind's opam file to have it work (until now :))

setenv: [
  [OCAMLFIND_CONF = "/dev/null"]
  [OCAMLPATH = "%{lib}%"]
  [OCAMLFIND_DESTDIR = "%{lib}%"]
  [OCAMLLIB = "%{ocaml:lib}%"]
]

Of course, this is opam 2.0 only. Note that the variables defined this way will also be exported by opam env.

[dbuenzli]

More generally what do you think the strategy should be with programs that have configuration files in etc ? E.g for odig should I simply add:

setenv: [
   ODIG_CONF="%{etc}%/odig.conf"
]

to the opam file ?

[AltGr]

This is a good fallback, but it would be better if we could avoid cluttering the user environment. Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ? Failing that, maybe you could rely on $OPAM_SWITCH_PREFIX/etc (opam 2 will always define that variable as part of opam env).

EDIT: The difference between the two is that you get the switch as set in the environment (the two may not be in sync if eval $(opam env) wasn't called; for example if you just cd to a dir where there is a local switch)

[dbuenzli]

Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ?

That doesn't feel a very good solution. While it would be ok for odig to do so given its scope, there are many programs out there for which it would be absurd to call opam at runtime.

[AltGr]

Note: setting OCAMLLIB = "%{ocaml:lib}%" is a bad idea at the moment. This is due to a limitation of opam when it reverts changes to environment variables: when you switch away from the switch where this was set, assuming that the new switch doesn't set it, rather than undefine the variable, opam will set it to the empty string. And ocamlc doesn't work properly with OCAMLLIB="".

[dbuenzli]

Re my own "problems" I'm wondering if there might be a standard unix env var that distribution have agreed upon that allows to specify a path to configuration (a little bit of search on the web didn't reveal anything so far).

More generally I wonder if @gasche's work on reproducible builds and that BUILD_PATH_PREFIX_MAP specification might be useful in general for the system (so that the build products do not depend in the switch you compiled them, but I don't know exactly which strategy you adopted).

[gasche]

It's a nice coincidence that I spent most of my day today so far, at the Mirage retreat, implementing an OCaml library to deal with BUILD_PATH_PREFIX_MAP (encoding, decoding, rewriting), that I hope to release tonight. (My goal is to propose the code for inclusion in the compiler codebase to make absolute paths in the .cmo debug information resilient to build-path changes, but I'm releasing an external library with the hope to let users use it in their projects.)

[dbuenzli]

@gasche Very cool. Please don't forget about the .cmis and the new --keep-locs default.

[gasche]

A pre-release of the library is available at

https://gitlab.com/gasche/build_path_prefix_map/

I'd like to test its usage in the OCaml compiler in practice (even only in a PR) before making a release.

type path = string
type path_prefix = string
type error_message = string

val encode_prefix : path_prefix -> string
val decode_prefix : string -> (path_prefix, error_message) result

type pair = { target: path_prefix; source : path_prefix }

val encode_pair : pair -> string
val decode_pair : string -> (pair, error_message) result

type map = pair option list

val encode_map : map -> string
val decode_map : string -> (map, error_message) result

val rewrite_opt : map -> path -> path option
(** [rewrite_opt map path] tries to find a source in [map]
    that is a prefix of the input [path]. If it succeeds,
    it replaces this prefix with the corresponding target.
    If it fails, it just returns [None]. *)

val rewrite : map -> path -> path

[AltGr]

Just a quick note: even with OCAMLPATH, OCAMLLIB, OCAMLFIND_CONF properly defined, ocamlbuild -where returns the wrong result. There seems to be no override in this case. The code refers to https://github.com/ocaml/ocamlbuild/issues/69

[gasche]

The "new" heuristic that is implemented in ocamlbuild is described in https://github.com/ocaml/ocamlbuild/pull/240/files ; it looks at the variable OCAMLLIB dynamically, and at the configure-time values OCAML_LIBDIR and OCAMLBUILD_LIBDIR -- in particular, OCAMLLIB is only used if OCAMLBUILD_LIBDIR is a sub-path of OCAML_LIBDIR, and in that case the returned path is OCAMLLIB + (OCAMLBUILD_LIBDIR - OCAML_LIBDIR).

You should be able to reproduce the fact that, if those variables were properly set up at ocamlbuild-configuration time, OCAMLLIB can be used to influence the setting. For example, on my machine, with a completely standard opam-installation of ocamlbuild (0.11.0 on 4.06.0), I observe:

$ OCAMLLIB='foo/' ocamlbuild -where
foo/ocamlbuild

I am not sure what your setup is, and what is the result you expect, but it looks like moving an ocamlbuild binary from one opam-user to another should work as long that the filesystem organization within the OPAM root is the same.

I guess that maybe the problem is that you want to setup OCAMLLIB to something else than the opam root directory. In that case, This is a hack, but is there a configure-time choice of OCAML_LIBDIR and OCAMLBUILD_LIBDIR such that the difference correctly goes from ocamlc -where to ocamlbuild -where after relocation?

(Of course I would be happy to accept changes to the heuristic if they preserve the current use-cases and make your life easier.)

[AltGr]

Thanks for the details. I am testing a binary cache of opam packages, i.e. assuming that I can get the installed artefacts of one switch, and put them into another.

In this specific case, I had installed the packages first in a temporary switch /tmp/foo/_opam, then the caching system recovered their installed files and put them into ~/.opam/4.05.0. The above patch to ocamlfind's opam file had been applied, but maybe resulted in a confusing setup that broke ocamlbuild's configuration ? I believe it was the case that ocamlbuild was not taken from cache, but rebuilt on the new switch, against the relocated ocamlfind, and with the variables defined as such:

OCAMLFIND_CONF = "/dev/null"
OCAMLPATH = "~/.opam/4.05.0/lib"
OCAMLFIND_DESTDIR = "~/.opam/4.05.0/lib"
OCAMLLIB = "~/.opam/4.05.0/lib/ocaml"

[gasche]

Could you compile ocamlbuild with --keep-build-dir and get the content of Makefile.config in the build directory? This file records the configure-time value of OCAML_LIBDIR, and a variable LIBDIR that corresponds to what I called OCAMLBUILD_LIBDIR in my message above.

[AltGr]

I have been trying to, but hitting other issues along the way. I'll get back to it asap :)

[Pitometsu]

why just do not use the Nix environment for that?

[Lupus]

BUILD_PATH_PREFIX_MAP MR is merged, isn't that feature sufficient to solve relocation? opam-bin-cache.sh just needs to maintain this prefix map and export it down to the individual package building environment? I've tried the bin cache script myself as a way to speedup creation of multiple local switches, and things got broken with ocamlfind after I wiped one of the switches and created it again. When it works, speedup looks really great! Would be awesome to see relocation solved.

[Khady]

Please don't close this one dear bot.

[github-actions[bot]]

This issue has been open 90 days with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. If you come across this issue in the future, you may also find it helpful to visit our forum at https://discuss.ocaml.org where queries related to OCaml package management are very welcome.

[gasche]

I guess now this is a good opportunity to ask @dra27 about the current state of his relocation-support effort.