Lock feature is restricting a lot of things on the opam repository

[hannesm]

Dear Opam development team,

the opam repository -- a git repository consisting of a lot of packages -- underlies various restrictions since "it may break lock files". These restrictions include some version boundaries, marking packages installable, ... (and are not well documented as far as I can see, neither are automatically checked).

From my perspective, there are already existing code paths in opam to allow "distributing a reproducible environment" -- (namely opam switch export --full --freeze), and the lock feature could learn from it (please also note that opam depends on some system packages, and these need to be captured as well (since they may break in versions) -- take a look at https://github.com/roburio/orb/ or a fully featured reproducible utility (including system packages, environment variables, opam switch export) -- and https://builds.robur.coop for this utility in action).

Now, for opam lock, I'm still not entirely sure what the goal is. Is it "roughly the same environment" or "no installation failures on a second system"? Depending on the goal, either embedding the opam metadata in a lock file (maybe it evolves to a switch export then? -- and would not depend on the opam repository), or at least embedding the opam-repository commit hash (if you can require all repositories to use git) would be sensible... but as mentioned, for binary reproducibility and for "no installation failure" you'll need to track the system packages as well.

It would be nice to document "what is allowed" and "what is forbidden" in opam-repository, and taking common workflows into account (packagers sometimes make errors and need to fix things, i.e. retract packages), or to revise opam lock to be less dependent on opam-repository.

Please bear with me if I missed documentation and the precise goal of opam lock. As a package maintainer I find it a feature that restricts my ability to move forward, and makes me doubtful about publishing packages to the main opam repository.

[rjbou]

opam lock is meant / was designed to share a development setup: I know that with these libraries versions my program/package compiles, I want to snapshot them to share them to someone else. So yes, it is "roughly the same environment", it's not as complete as switch export, is not meant for reproducibility (in the meaning of reproducible builds) ; and it shouldn't be used for that. It is meant to be installable simply in a already present (current dev) switch. That's why the format is simply an opam file : you can set the lock file as your opam file if wanted. We could add information, and it will benefit for development setup reproducibility to have the git repository & its hash (if possible, a default user wouldn't have the information). To integrate that, it will need a complete review of some part of opam's functioning: add repositories à la volée, how/when use them: tie package with its repo? what if two lock files have the same dependency but different hashes / repositories, etc. Agree that some of these questions (and others) can even be asked for opam lock current state, and have no satisfying answer.

For opam repository, maybe better see with repository maintainers about the guidelines / redirect to the good page.

On lock file restricting opam repository, i'm afraid it's not really the case. From what I know, packages shouldn't be removed to avoid having install issues (removed packages), and these install issues can originate from a lock file. Or from any other opam file. Or script/tutorial.

[hannesm]

ok, thanks for your answer and pointing to the written specification. Closing this since I doubt there will be any action.