MPR#4870: Ocamlbuild is too slow when using the "include" tag on many directories

[damiendoligez]

PR transferred from https://caml.inria.fr/mantis/view.php?id=4870 [original reporter: @AltGr]

when there are lots of directories in your project, using "include" as an easy way to compile (given you don't have clashes between module names) leads to a very long time just to check what should be done: we got to more than 20 seconds in case there is nothing to do.

---

It seems that every rule is duplicated for every possible directory, which may be at the root of the problem.

We could circumvent it by having a cleaner hierarchy: never use the tag "include", but build and link libraries instead, and use Pathname.define_context by hand for directories that need to work together.

Also, we had to add:

        dep ["use_"^lib; "byte"] [lib^".cma"];
        dep ["use_"^lib; "native"] [lib^".cmxa"];

for the internal libraries to be built correctly.

[whitequark]

Triage: what is to be done here?

[whitequark]

This seems like it would be fixed by https://github.com/ocaml/ocamlbuild/issues/188, but I'm not sure.

[gasche]

Frankly, I don't know what should be done. These days (and in the recent past) I had zero time to work on ocamlbuild's performance, which are "good enough" for most people that enjoy it's convenience for small-enough project and "disappointing" for demanding users which are free to use another tool of their choice. I agree that making define_context easier to use could be part of the solution in this particular case (and that include is too heavy-handed).

For performance in general, I have four ideas:

• push on the monadic-rules for parallelization work that I started in 2013 (blog post)
• work on a fast way to rebuild a project assuming that the dependencies have not changed (we would store a "build recipe" generated by the log of a correct build, and have a simple and fast tool to reproduce the actions in the build plan with maximal parallelism, and defer to ocamlbuild in case of a failure due to dependency change); this could be useful for people distributing ocamlbuild-using packages in particular
• consider the use of codept within ocamlbuild; at any point where we have some high-level view of a set of modules that have to be compiled, we could use codept to compile the global intra dependencies of all these modules as once, and improve the build plan (such logic would be related to what Ocaml_compiler.prepare_link is doing today); this is not so easy because it requires to cut through the abstraction layer in a way that is not natural for ocamlbuild's logic (a priori when you have a .mllib module, there is no reason to assume that all compilation units will materialize as .ml files at some point in the build, they could just as well be object files in the user repository or a different way to produce them)
• we should investigate the use of modification times before hashes as a way to cut down the pre-build and nothing-to-do times

Longer-term ideas include relying on some other people's performance-oriented work (eg. jenga/jbuilder), but for those the horizon of me personally having the time to work on it seems even farther away (I don't think this can be conceived as a weekend-hack project, while some of the above can), and then there is the question of just becoming a frontend to some other tool (which ironically means keeping the more ugly/fragile of ocamlbuild alive and cutting the arguably nicer internals, but if that is what is best for users there is no reason not to).