Nested workspaces

[emillon]

Dune is built on the concepts of projects and workspaces. Projects now always have a dune-project file, workspaces can be identified by the presence of a dune-workspace file but not always.

Composability is an important feature: putting several projects next to each other in the same workspace make them visible. Various source files are attached to the right project through the hierarchy of dune-project files.

But workspaces are different: they correspond to a closed-world usage of dune where the environment is determined, by opposition to the open-world nature of projects which are to be used in various environments (several versions of the dependencies, etc). So: workspaces are not compositional because they compete as the source of truth.

This brings the question about what we should do when a dune-workspace file is found in the hierarchy in an existing workspace. 2 cases exist, depending on whether the outer workspace is explicit (dune-workspace file where dune is invoked) or implicit (dune-project file with no dune-workspace file where dune is invoked).

It seems that the current behavior is to ignore the file. We could error out, warn, or even ignore the whole subtree in addition.

This can happen when public projects that contain a dune-workspace file are grafted into a monorepo. This is innocuous when installing through opam, and there are valid use cases for a workspace/opam combination (for example, CI uses opam files to get dependency), but it could be worth adding a check on opam-repo-ci if we go that way.

[snowleopard]

Let me list two specific issues we've encountered in Jane Street's monorepo, where we have a dune-workspace file at the monorepo's root.

• If there is <root>/some/path/dune-workspace then cd <root>; dune build and cd <root>/some/path; dune build give different results: the first build uses <root>/dune-workspace and the second <root>/some/path/dune-workspace.

• If there is <root>/some/path/dune-project, then cd <root>; dune build may end up using inconsistent build settings across the monorepo, in particular, if <root>/some/path/dune-project sets a different dune lang, then files under this directory may be built incorrectly. For example, they may have incorrect build path prefix map: https://github.com/ocaml/dune/blob/71a0d184e1ed546b6046c309e39f9e3ca807c986/src/dune_engine/execution_parameters.ml#L89

[nojb]

At LexiFi we are in the same boat as what @snowleopard describes.

Wondering out loud: perhaps a completely different file dune-monorepo with the right "monorepo" semantics (ie override settings in all nested dune-project files and ignore nested dune-workspace files) could be a way forward?

Alternatively, give these semantics to the existing dune-workspace files? (but we would need to decide on a way that Dune could use to figure out what is the "current" dune-workspace file it should follow).

[rgrinberg]

As discussed in the slack chat, it's always permitted to set workspace files directly in the command line or via an environment variable.

Additionally, we discussed reverting to the behavior of letting the topmost workspace file take precedence. Previously, this behavior was frowned upon b/c we created dune-project files automatically, but now I don't see anything wrong with it.

[snowleopard]

At LexiFi we are in the same boat as what @snowleopard describes.

@nojb Cool, good to know that!

Wondering out loud: perhaps a completely different file dune-monorepo with the right "monorepo" semantics (ie override settings in all nested dune-project files and ignore nested dune-workspace files) could be a way forward?

I think it's sufficient to add a setting like (monorepo_semantics true) to dune_workspace. No need for yet another file.

Additionally, we discussed reverting to the behavior of letting the topmost workspace file take precedence. Previously, this behavior was frowned upon b/c we created dune-project files automatically, but now I don't see anything wrong with it.

@rgrinberg That sounds good to me too. Happy to review a PR (or PRs) that implement this idea!