Improve UX around portable lockdirs

[gridbugs]

trafficstars

Using this issue to collect cases of UX confusion around portable lockdirs.

When solving dependencies dune prints all packages in the solution, which may include multiple different versions of the same package

For example:

- sexp_pretty.v0.15.1
- sexp_pretty.v0.17.0
- sexplib.v0.15.1
- sexplib.v0.17.0
- sexplib0.v0.15.1
- sexplib0.v0.17.0
- spawn.v0.17.0
- splittable_random.v0.15.0
- splittable_random.v0.17.0
- stdio.v0.15.0
- stdio.v0.17.0
- stdlib-shims.0.3.0
- system-mingw.1
- textutils.v0.15.0
- textutils.v0.17.0
- textutils_kernel.v0.15.0

Only one version of each package will be built on any given platform, but different platforms may require different versions of the same package. One possibility for making this less confusing would be to annotate packages in the output of dune pkg lock with the condition under which they will be chosen if they aren't chosen on all platforms.

[rgrinberg]

I discussed this with @maiste and @art-w offline and issue with the current implementation is that multiple versions for the same package are going to be selected far more often than the user expects. It is much more common for the user to want the following additional constraint:

If a package is selected for more than one configuration, the solver should try to satisfy both configurations with the same package version.

Or put another way, multiple versions of a package should be opt-in and never offered by default.

We've discussed how we might want to express this, and it seems like we'd really want to express this constraint in the solver and only solve a single SAT problem for the all configurations at once. I don't have concrete ideas how to do this so far, but this direction seems interesting to me.

[gridbugs]

we'd really want to express this constraint in the solver and only solve a single SAT problem for the all configurations at once

I couldn't agree more. Solving a single SAT problem rather than one problem per platform is the better option for many reasons. The choice to implement it with one problem per platform came from internal discussions at Tarides where we concluded that we'd do it this way first as it's easier/faster to implement, but format lockfiles in such a way that the solver algorithm can be changed at a later time.

[gridbugs]

Regarding the original issue of Dune printing all the deps in one list, leading to confusion when a single package appears multiple times with different versions, I'm working on an alternative way of printing the dependencies when portable lockdirs is enabled which should make this clearer. Dependencies common to all platforms are printed separately from those which are platform-specific, and each supported platform with specific dependencies gets printed along with the dependencies only installed on that platform.

Here's an example:

$ dune pkg lock
Solution for dune.lock

This solution is supports the following platforms:
- arch = arm64; os = linux
- arch = arm64; os = macos
- arch = arm64; os = win32
- arch = x86_64; os = linux
- arch = x86_64; os = macos
- arch = x86_64; os = win32

Dependencies on all supported platforms:
- base.v0.17.3
- base-unix.base
- csexp.1.5.2
- dune-configurator.3.20.2
- jane-street-headers.v0.17.0
- jst-config.v0.17.0
- ocaml.5.3.0
- ocaml-base-compiler.5.3.0
- ocaml-compiler.5.3.0
- ocaml-compiler-libs.v0.17.0
- ocaml-config.3
- ocaml_intrinsics_kernel.v0.17.1
- ppx_assert.v0.17.0
- ppx_base.v0.17.0
- ppx_cold.v0.17.0
- ppx_compare.v0.17.0
- ppx_derivers.1.2.1
- ppx_enumerate.v0.17.0
- ppx_expect.v0.17.3
- ppx_globalize.v0.17.2
- ppx_hash.v0.17.0
- ppx_here.v0.17.0
- ppx_inline_test.v0.17.1
- ppx_optcomp.v0.17.1
- ppx_sexp_conv.v0.17.1
- ppxlib.0.37.0
- ppxlib_jane.v0.17.4
- sexplib0.v0.17.0
- stdio.v0.17.0
- stdlib-shims.0.3.0
- time_now.v0.17.0

Additionally, some packages will only be built on specific platforms.

arch = arm64; os = macos:
- cmdliner.1.3.0

arch = arm64; os = win32:
- cmdliner.2.0.0
- conf-mingw-w64-gcc-i686.1
- flexdll.0.44
- ocaml-env-mingw32.1
- system-mingw.1

arch = x86_64; os = macos:
- cmdliner.1.3.0

arch = x86_64; os = win32:
- arch-x86_64.1
- cmdliner.2.0.0
- conf-mingw-w64-gcc-x86_64.1
- flexdll.0.44
- ocaml-env-mingw64.1
- system-mingw.1

I think this addresses the primary usability issue with portable lockdirs, which is that users were confused about why multiple versions of the same package are included in a solution.

[gridbugs]

It's possible to take this further and make it an error if a solution contains multiple versions of the same package (possibly surpressed by some configuration). Either way I think clarifying which platforms are included in the solution and which packages are specific to those platforms is a better message to print after solving, so I'll just start with changing the message.

[rgrinberg]

It's possible to take this further and make it an error if a solution contains multiple versions of the same package (possibly surpressed by some configuration)

Given that we can't guide the solver towards such a solution, this is probably premature. Stay tuned though, there might be a way forward with a new solver.

[gridbugs]

In the meantime, how do you feel about enabling portable lockdirs by default once https://github.com/ocaml/dune/pull/12620 is merged?

[rgrinberg]

What would be the default set of problems that a user would solve for?

[gridbugs]

It will solve for platforms Linux (x64, arm64), MacOS (x64, arm64) and Windows x64.

[rgrinberg]

Which set of distros does that include? Anyhow, It's a reasonable set of platforms but I'm not sure how good default it would be. Would it be too intrusive to just ask the user to specify the set of platforms they intend to support?

EDIT: Actually, this set of platforms isn't reasonable. There's no need to include windows on this list until package management can actually build things on Windows.

[gridbugs]

Which set of distros does that include?

The default platforms don't specify distros, though this can be configured in the custom list of platforms in dune-workspace. If someone needs a package solution which is distro specific they're expected to add that platform to their dune-workspace. Note that this doesn't cause problems for depexts because lockfiles contain a copy of the entire depext section from the opam file for its package.

Actually, this set of platforms isn't reasonable. There's no need to include windows on this list until package management can actually build things on Windows.

Ok that's fair, I can remove windows from the set of default platforms.

[Alizter]

I believe this issue has almost been fully addressed. There is still however the remaining question of which default set of platforms do we target.

Let me recall the proposals thusfar (please feel free to expand on these points):

1. Target the platform of the user running the lock

Advantages:

• We only solve once
• Works out of the box for each platform

Disadvantages

• Expanding the platform set has to be done by configuring it manually
• Not immediately shareable in a repo with the guarantee it will work on the platforms we are interested in.

2. Target a predefined default set of platforms

Advantages:

• Works out of the box for each supported platform
• Everybody produces the same lock file

Disadvantages:

• We solve multiple times (though could be combined into single SAT or use different solver)
• Might not work for platforms not in the target platform, even if they are locking

3. Mutate the list of supported platforms for each new platform

Advantages:

• Works out of the box for each platform
• Platforms targeted grow as potential users grow

Disadvantages:

• Unintuitive
• Not reproducible across platforms
• Relies on order
• Not everyone wants random platforms appended

---

I suggested 3 a few meetings back, but I don't like it anymore. My own feeling has been between 1 and 2. Right now, 1 seems sensible.

[shonfeder]

(3) seems like the best option, IMO.

Addressing the list of supposed disavantages,

• What is unintuitive about it? It fits my intuition.
• In what sense is it "Not reproducible across platforms"? IIUC, each platform should be able to reproduce the locked dependency plan checked in for the platform. If there's isn't one, it can add a new solution. Relocking is not "reproducible" generally in any case, unless all the inputs are fixed (e.g., opam repo has, same platform etc.). So what is not provided here?
• What does "Relies on order" mean?
• "Not everyone wants random platforms appended": I don't understand this either. Why would random platforms be appended?

Regarding the other two options:

(1) seems to miss the whole point of the portable format, since we could already do this without the portable format. "Not immediately shareable in a repo with the guarantee it will work on the platforms we are interested in." this seems like it should be avoided as much as possible. Requiring esoteric configurations to build a project on each new platform when we have an automated solution available seems to have upside for users, afaict.

(2) Is preferable to (1), in that it is at least what we already have (so requires no extra work), and it is compatible with (3). But I understand the critiques here and agree that (3) is preferable in its simplicity, and in that locks then certify solvability (and if they are checked in, buildability) on actual platforms.

[rgrinberg]

3. seems to be equivalent to the following steps:

4. Adding a new configuration the the list of valid configurations

5. Generating a new solution

It could be that it is worth automating, but it doesn't seem like it would have a high payoff. In addition, it would get in the way for all folks trying to run the build plan in an explicitly unsupported configuration, as it would modify committed files.

Here's an alternative approach that attempts to identify two use cases that we'd like to support. The first use case is a committed lock directory for a set of supported configurations. The second use case is a not committed lock directory generated on the fly. Theses are not the only use case we intend to support, but just the ones for which we have a happy path for.

1. First, we lean on the fact that we have auto-locking. Since lock directories can be generated on the fly, the main reason to have lock directory in the source is to commit and reuse it. All use cases that do not want to commit the lock directories but still generate it manually are not our focus. Though they are still supported by the user explicitly pointing dune to a lock directory.

2. $ dune pkg lock only generates for the configurations present in the workspace file. If there's no workspace file, it will create one. There's really no avoiding the creation of a workspace file if the lock directory is to be committed. This workspace file is needed for everybody to reproduce a solution for this configuration if it is to be checked in.

3. dune build will use the lock directories if the following 2 conditions are met: the lock directory exists and contains a configuration that matches the user's environment. Otherwise, we fallback to auto-locking with possibly a warning. The user of course still has the option to add their own configuration and regenerate, or to generate their private lock dir using their own workspace file. These are all possible, but aren't core and therefore require a bit more configuration.

It might help how we'd spell out how we'd use these features in dune.

1. A workspace file with 3 configurations (mac, linux, windows) would be committed along with a solution for all 3 platforms.
2. All 3 configurations would be built and tested in CI.
3. No additional configuration would generally be allowed to be added.
4. Anybody developing on an unsupported platform would be encouraged to rely on auto-locking.

[Leonidas-from-XIV]

Now with autolocking & copy rules I think we can have our cake and eat it too, so to say. Here's my proposal.

Implicit platforms for autolocking, explicit platforms for source lock files

1. If there's no lock dir, autolock for the current platform only. This is fast and easy, works out of the box on every platform. No issue with sharing because these lock dirs are never shared.
2. If there is a lock dir in the source, try to use that. Error out if there's no matching lock dir (as that signals that the repo is not supporting your platform (yet)), with a hint of how to generate a lock dir for your platform.
3. Lock dirs in source are created by dune pkg lock (or some other command). Make dune pkg lock require a set of platforms to lock for in the dune-workspace (or somewhere else). That way whoever creates a portable lock dir explicitly specifies a set of supported platforms for the project in a configuration file. And if the set is to be extended, then the set gets expanded and future dune pkg lock invocations will generate the new platforms.

Advantages:

1. Works out of the box on any platform unless explicitly changed.
2. If there's a subset of supported platforms, they are defined in a canonical config file

Disadvantages:

1. A bit annoying workflow difference between lock files and autolocking.

[Leonidas-from-XIV]

As @rgrinberg posted the proposal at nearly the same time as me and the proposals only differ in the fact that Rudi suggests to use autolocking (with a warning about ignoring the lock dir in source) I think I actually prefer that approach to erroring out. That makes dune pkg work on unsupported platforms, instead of requiring users on such platforms to keep a custom dune-workspace.