refactor: improve findlib loading performance

[rgrinberg]

Do not call [Path.stat] for every single package. Just build an index of packages for every directory.

Signed-off-by: Rudi Grinberg [email protected]

[talex5]

Also, opamStd.ml's terminal_columns installs a signal handler that uses lazy (and isn't atomic): https://github.com/ocaml/opam/blob/d1fe3e82b4889c7f24b83e3284720b3611e93574/src/core/opamStd.ml#L958-L961

This crashes under multicore if two threads try to log at the same time:

CamlinternalLazy.Undefined
Raised at CamlinternalLazy.force_gen_lazy_block in file "camlinternalLazy.ml", line 75, characters 9-24
Called from OpamStd.OpamFormat.reformat in file "src/core/opamStd.ml" (inlined), line 1280, characters 44-71
Called from OpamConsole.warning.(fun) in file "src/core/opamConsole.ml", line 584, characters 6-63

[avsm]

The segv is possibly due to the Stdlib.Parsing module being thread-unsafe and involving C bindings (https://github.com/ocaml/ocaml/blob/trunk/stdlib/parsing.ml). A standalone testcase would be suitable to create an issue on the ocaml/ocaml repo.

[kit-ty-kate]

I had a shot at debugging this issue. The main problem is actually ocamlyacc that is not thread-safe (not just not domain-safe). Adding a mutex to opam-file-format makes it work but is fairly ugly. The OCaml manual encourages to use menhir instead.

[kit-ty-kate]

Some update: this was partially fixed in opam-file-format.2.2.0 (https://github.com/ocaml/opam-file-format/pull/60) integrated in opam 2.4. There is larger move toward making the opam internals domain-safe (https://github.com/ocaml/opam/pull/5966, https://github.com/ocaml/opam/pull/5877) but it's quite a large non-critical work and it'll take us some time, but it's definitely happening as part of the future migration to OCaml 5