ocaml
Please keep around old versions of dune
It seems old versions of dune are now being (automatically?) removed from the repo. This has caused us some head-scratching and even now that we know this is happening, has adverse side-effects for our CI setup: we'd like to pin an old version of dune, since dune gets updated a lot and each time it does, the world needs to be rebuilt, thus costing a lot of CI time. (Or, at least coq/rocq needs to be rebuilt, which is the one thing we actually use opam for.) But with old versions of dune being pruned from the repo, this has now become harder -- we may have to resort to copying the dune package definition into our own repo or so, to avoid these slow rebuilds.
Dear @RalfJung,
thanks for opening this issue. For now, we restored the dune 3.17.2 package in the opam-repository, so your builds will succeed again. That version of dune is also marked to not being removed from the opam-repository, so subsequent archivals based on the maintenance-intent will leave it untouched.
There is still a question in my mind: The archival was announced on https://discuss.ocaml.org/t/opam-repository-archival-phase-3/ - but it seems this didn't catch your attention. What could be done to improve this? An idea from @raphael-proust is that we should include the steps needed to test the opam-repository after archival in the announcement (would that catch your attention, would you spend time to test it?). Should we announce such archivals elsewhere (for example on the caml-list?)? Should there be tooling taking a .opam.locked file and adding the x-maintained field to these (including a comment about where it is used)?
The idea behind the archival process is to automatically (based on the x-maintenance-intent) archive every few months opam packages, but of course we would like to not break any of your CI systems (see https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md for the plan, based on the discussions in https://github.com/ocaml/opam-repository/issues/23789 and various video meetings). So we're curious what can we do to improve the communication? We already planned for the possibility to mark packages with x-maintained: true so they're not being removed (for the exact use case, that these are used by external/private organizations/CI systems that pin to specific versions.
In the meantime, how to fix this:
- One path is to add the archive repository to your systems (but of course then you won't benefit from reduced solver times) (opam repo add archive git+https://github.com/ocaml/opam-repository-archive);
- another is to pin to a specific commit of the opam-repository (opam repo set-url default git+https://github.com/ocaml/opam-repository.git#$commit-hash);
- or take the time to update your lock file / version pins.
//cc @liyishuai and @Lysxia who worked on the rocq issue; also @7h3kk1d and @cyrus- for hazel, and @Blaisorblade for their internal systems, @LasseBlaauwbroek and @bertrand-sb may as well have input
Should we announce such archivals elsewhere (for example on the caml-list?)?
If you want the announcements to be noticed by the Rocq community, the best choice would be to announce them as well on the Rocq Discourse (https://discourse.rocq-prover.org/). That being said, I don't know if we would have realized the impact on the Docker-Rocq images.
The maintainers of Docker-Rocq are @erikmd and @himito, so perhaps they will be able to tell which solution they prefer.
Hello, it's not a big deal overall. I think I understand the rationale behind the change here and I now have a workaround. That being said, maybe I can bring some perspective as a user. By the way, obviously I'm very thankful for the work all the maintainers are doing to keep the repository working and make it better.
The update caught us off guard. Our CI relies on a few outdated packages, one of which is dune-configurator < 0.17.0. The reason is mainly for compatibility with older build environments like RHEL 8, but mostly engineering resources that would be required to upgrade that we don't currently have. Of course, ideally we wouldn't rely on those older versions, but that's the reality I live in.
As a user, the breakage was unexpected in two ways:
- No easy-to-find announcement. I can't really keep up with what's going on in https://discuss.ocaml.org/ unfortunately.
- To this day, there doesn't seem to be easy-to-find official instructions on how to download those older versions of OCaml packages.
- In general, it's unusual for versions of packages to disappear from a package registry. When it happens (e.g. in Python or Rust), it's usually for security reasons (e.g. a malicious package) or because the author pushed a broken version, so they yank it from the registry to push another version. But never have I seen such archiving being done in any other registry before.
- I'm not suggesting the opam registry should work differently, but rather highlighting that users, especially those familiar with other languages, may be even more surprised by the archival.
With that said, maybe some of these could improve the situation:
- An announcement and workaround on https://opam.ocaml.org/blog/.
- Official instructions on how to access older packages (in the manual? or built in opam?)
- And I mean user instructions, not package maintainer instructions.
- More ambitious: In the future, keep more packages in the main registry but somehow reduce their impact on opam resolution and OCaml CI?
- Really unsure about that. I just know that the quality of resolution and tests in the registry is quite high and thus costly, but maybe we don't need that quality for 100% of packages.
(It would have been ideal if opam could have warned us ahead of time that the packages we depend on are about to be removed from the registry, but that would have been non-trivial to implement in opam and it's too late for that anyway.)
I hope this helps!
Thanks for the long and thoughtful reply!
There is still a question in my mind: The archival was announced on https://discuss.ocaml.org/t/opam-repository-archival-phase-3/ - but it seems this didn't catch your attention. What could be done to improve this?
That's a very fair question. I have basically no contact with the ocaml community and do not frequent the ocaml forum or mailing list, I am just a Rocq user. However, I do happen to have https://opam.ocaml.org/blog/feed.xml subscribed, so an announcement there would have gotten my attention.
But this may be an interesting data point for you: there is some number of opam users that are not ocaml users.
What would have aided in debugging is if opam didn't just say
"[ERROR] Package dune has no known version 3.13.1 in the repositories"
but somehow pointed out that the version has been archived, with a url providing context. I don't know whether there is a practical way to achieve this.
(Due to the way our CI works, the command that fails this way is an opam pin of the specific version. So it may also be the case that there would have been a better error if the missing version would have surfaced in a different way, e.g. via a missing dependency.)
That being said, I don't know if we would have realized the impact on the Docker-Rocq images.
To be clear, we are not using the docker-rocq images, we have our own CI. I was not aware that docker-rocq was also affected -- in fact, I was thinking maybe we should finally migrate to docker-rocq to avoid such issues...
In general, it's unusual for versions of packages to disappear from a package registry. When it happens (e.g. in Python or Rust), it's usually for security reasons (e.g. a malicious package) or because the author pushed a broken version, so they yank it from the registry to push another version. But never have I seen such archiving being done in any other registry before.
Yes, this has probably also contributed to my initial confusion. Also, the versions still showed up locally after they were missing on CI, and I thought I had done an opam update, but maybe I did not, or maybe the update took a bit longer to propagate to my local mirror if that's a thing (I am traveling and hence geographically quite far away from the CI machine right now).
For now we've just added the archive repository.
There conflicts due to the archived packages that seem to require maintainer help to resolve. e.g. ppx_deriving_qcheck requires ppx_lib >= 0.36.0 while ppx_js_style requires ppx_lib < 0.36.0. In the archive we use ppx_deriving_qcheck 0.6 which allows for ppx_lib < 0.36.0. By only keeping the most recent versions of packages there doesn't seem to be enough time for upates to propagate through the system like this.
The dependency resolution algorithm was also a bit noisy to track down the actual issue. It seemed to eagerly lock down some versions making the actual conflict hard to spot.
opam install output
[ERROR] Package conflict!
* No agreement on the version of ocaml-base-compiler:
- (invariant) → ocaml-base-compiler = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-base-compiler (< 3.07+1 | = 3.08.0 | = 3.08.1
| = 3.08.2 | = 3.08.3 | = 3.08.4 | = 3.09.0 | = 3.09.1 | = 3.09.2 | =
3.09.3 | = 3.10.0 | = 3.10.1 | = 3.10.2 | = 3.11.0 | = 3.11.1 | = 3.11.2 |
= 3.12.0 | = 3.12.1 | = 4.00.0 | = 4.00.1 | = 4.01.0 | = 4.02.0 | = 4.02.1
| = 4.02.2 | = 4.02.3 | = 4.04.0 | = 4.04.1 | = 4.04.2 | = 4.05.0 | =
4.06.0 | = 4.06.1 | = 4.07.0 | = 4.07.1 | = 4.08.0 | = 4.08.1 | = 4.09.0 |
= 4.09.1)
You can temporarily relax the switch invariant with `--update-invariant'
* No agreement on the version of ocaml:
- (invariant) → ocaml-base-compiler = 5.2.0 → ocaml = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml < 4.09.0
* No agreement on the version of ocaml-base-compiler:
- (invariant) → ocaml-base-compiler = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → ocaml-base-compiler (=
4.10.0 | = 4.10.1 | = 4.10.2 | = 4.11.0 | = 4.11.1 | = 4.11.2)
* No agreement on the version of ppxlib:
- deps-of-hazel → ppx_deriving_qcheck → ppx_deriving >= 6.1.0 → ppxlib >=
0.36.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0
* Incompatible packages:
- (invariant) → ocaml-base-compiler = 5.2.0
- (invariant) → ocaml-system = 5.2.0
* Incompatible packages:
- (invariant) → ocaml-system = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-base-compiler (< 3.07+1 | = 3.08.0 | = 3.08.1
| = 3.08.2 | = 3.08.3 | = 3.08.4 | = 3.09.0 | = 3.09.1 | = 3.09.2 | =
3.09.3 | = 3.10.0 | = 3.10.1 | = 3.10.2 | = 3.11.0 | = 3.11.1 | = 3.11.2 |
= 3.12.0 | = 3.12.1 | = 4.00.0 | = 4.00.1 | = 4.01.0 | = 4.02.0 | = 4.02.1
| = 4.02.2 | = 4.02.3 | = 4.04.0 | = 4.04.1 | = 4.04.2 | = 4.05.0 | =
4.06.0 | = 4.06.1 | = 4.07.0 | = 4.07.1 | = 4.08.0 | = 4.08.1 | = 4.09.0 |
= 4.09.1)
* Incompatible packages:
- (invariant) → ocaml-base-compiler = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-variants
* Incompatible packages:
- (invariant) → ocaml-system = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-variants
* Incompatible packages:
- (invariant) → ocaml-base-compiler = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → dkml-base-compiler
* Incompatible packages:
- (invariant) → ocaml-system = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → dkml-base-compiler
* Incompatible packages:
- (invariant) → ocaml-system = 5.2.0
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → ocaml-base-compiler (=
4.10.0 | = 4.10.1 | = 4.10.2 | = 4.11.0 | = 4.11.1 | = 4.11.2)
* Missing dependency:
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml-migrate-parsetree → ocaml-variants (= 4.08.0+beta2 | =
4.08.0+beta3)
no matching version
* Missing dependency:
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-variants → ocaml-beta
unmet availability conditions: 'enable-ocaml-beta-repository'
* Missing dependency:
- deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
0.36.0 → ocaml < 4.10 → ocaml-variants → system-msvc
unmet availability conditions: 'os = "win32"'
No solution found, exiting
With regards to notification. It would be nice if we had a warning in our builds showing that some of the packages we depended on were up for archiving.
@RalfJung interesting -- so do you use another dune version you'd like to keep? I thought the rocq-base would be the thing you're using as well.
Should we announce such archivals elsewhere (for example on the caml-list?)?
If you want the announcements to be noticed by the Rocq community, the best choice would be to announce them as well on the Rocq Discourse (https://discourse.rocq-prover.org/). That being said, I don't know if we would have realized the impact on the Docker-Rocq images.
The maintainers of Docker-Rocq are @erikmd and @himito, so perhaps they will be able to tell which solution they prefer.
It's not too late to make such an announcement on the Rocq Discourse. I guess that many Rocq users are still not aware of the issue and will see similar issues in the coming weeks. Also, it would be nice if the announcement included a recommended way to fix it (either opening an issue to unarchive some packages or locally adding opam-repository-archive as an OPAM repository).
@pi8027 @Zimmi48 great idea - I just wrote a post at https://discourse.rocq-prover.org/t/announcement-opam-repository-opam-ocaml-org-archiving-process/
I ran into a similar issue this morning where a CI machine that tries to pin dune 3.6 failed. I had even seen the discourse post, but it's still not always obvious when you will be affected. In my case, the CI job has been happily running without changes in quite some time, so it was out of my mind.
My 2¢ is that automatic archiving of packages that primarily contribute executables might be distinct from "true" library packages, as it is infinitely easier for things like dockerfiles or CI jobs to depend on those without there being any trace in the opam repository. Of course, I'm not even sure if opam distinguishes these types of package
@7h3kk1d I see. so it seems there's no opam package on opam-repository that depends on both ppx_deriving_qcheck and ppx_js_style -- and thus the older ppx_deriving_qcheck got archived. what stops you from publishing hazel to opam-repository?
@WardBrian there's no such distinction in opam/opam-repository. was it hard to update your CI job?
@bertrand-sb thanks for your comment. It'd be great if we could have opam warning about soon-to-be-archived packages -- but unfortunately that is rather tricky (and will require some effort on the opam side, as well as a publish cycle, combined with distributions picking up the opam version). While it is tricky / will take some time, it is definitively a good idea.
Similar to @RalfJung suggestion that opam should print out "this package version has been archived" -- so far opam doesn't know anything about the archived repository.
was it hard to update your CI job?
Not terribly, no. To avoid needing to figure out exactly why we started pinning it (I vaguely recall we were having some issues building on intel Macs with old Apple SDKs and newer dunes, which is what this particular CI job would fall into), I've decided just to add the archive repo to that machine.
@7h3kk1d I see. so it seems there's no opam package on opam-repository that depends on both ppx_deriving_qcheck and ppx_js_style -- and thus the older ppx_deriving_qcheck got archived. what stops you from publishing hazel to opam-repository?
I don't think we considered it. I didn't realize it was idiomatic to publish tools built using ocaml to opam.
so do you use another dune version you'd like to keep? I thought the rocq-base would be the thing you're using as well.
We're using 3.13.1. We can't use 3.14 or newer since Coq 8.20 is not compatible with that and our infra is shared across all the Coq/Rocq versions we build with.
We've meanwhile just added dune 3.13.1 to our own opam repo so we're not blocked on this problem for now.
Hello, could you please reintroduce version 3.15.3 of Dune? We just got a bug report on the Rocq Platform repo that a released version of the Platform doesn't work anymore because of this. Maybe in the future, the Rocq Platform should pin the specific commit of the opam repo used for better reproducibility, though.
@Zimmi48 sure, we can do that. Would you mind to point to the place where dune 3.15.3 is referenced? In the linked issue and related PRs, I can see that you started to use the opam-repository-archive in some of your scripts (where, as you mention here as well, I'd suggest to pin the opam-repository to a specific commit) -- do you still need 3.15.3 back?
Thanks a lot for the PR. I have merged in the meantime, @Zimmi48 plase add the additional context directly on the PR as a comment
Thanks!
I will let the actual maintainers of the Rocq Platform (@MSoegtropIMC @Justme0606) answer this in detail. My request was so that the already released version of the Rocq Platform still works. For future versions, I think we can indeed adopt strategies for better reproducibility.
@hannesm: in Coq Platform I pin dune to a specific version because in the past we had a lot of trouble with incompatibilities and different packages in Coq Platform having non overlapping dune version restrictions. So I pick a version of dune and all Coq Platform contributors have to live with this one version - if there are real issues this is discussed then and maybe I change my pick, but pinning dune proved to be the only viable solution.
For 3.15.3 see https://github.com/rocq-prover/platform/blob/524ebcfe683873eff9f448259460d3380408bb0c/package_picks/package-pick-8.19~2024.10.sh#L52
Btw.: we are currently working on a PR to add the legacy opam repo to all legacy picks. In the future we expect that only the latest pick works without the legacy repo. The latest pick usually has a life time of about 9 months from early beta to release of next version.
In general, I suspect I'd rather disable the archive to create reproducible lock files, then use the archive together with lockfiles to bypass performance problems with solvers when installing packages.
@MSoegtropIMC it seems that currently, old dune releases will keep being removed (every 3 or 6 months), and since platform releases last longer, a new strategy will indeed be necessary. My experience is that upgrading dune can be a breaking change (even if the dune language is versioned, it can't keep perfect compatibility), so pinning dune is a good call for the Platform.
@Blaisorblade : thanks for confirming that we did the right choice with pinning dune.
IMHO we should discuss the policy of removing things after 3 months. The repo split was definitely a good thing, but IMHO we are getting from one extreme into another. I don't think there will be opam performance issues if packages are kept for a year.
If this policy is not negotiable, the only solution I see for the platform is to always add the legacy repo - which adverses the original intention of the repo split, because then we will have the same performance issues as before the split even with builds of latest SW.
@MSoegtropIMC Everything is negotiable. The policy of x-maintenance-intent is release based rather than time-based, so if you have a stable URL (or URLs)where we can grep for pinned (dune, but of course other packages as well) versions, we're happy to keep them in opam repository. We're then as well happy to archive them once your releases are out of maintenance (and thus the dune versions aren't required anymore).
An alternative I can see is that during a release you pin your opam-repository to a specific commit hash, and thus you will ever get the same package universe. This may obviously not be your intention if you like to receive updates (new releases) of opam/ocaml packages.
Is it possible that removing old versions of dune from the repo entirely breaks switches that have that version of dune? I saw this in a switch of mine:
[ERROR] No definition found for the following installed packages: { dune.3.17.2 }
This switch may need to be reinstalled
I was unable to do anything with that switch until I did opam update which brought that version of dune back into the definitions.
But it seems quite bad if I can't even update my switch to a new dune once the old dune it has gets removed from the repo...
Since you're all here, please allow me to direct you to the scheduled next archival phase at https://discuss.ocaml.org/t/opam-repository-archival-next-run-scheduled-2026-01-01 -- which includes instructions on how you can test your software with the branch of opam-repository where the packages would already be archived. This way, if you provide feedback the sooner rather than later, we can remove packages that you rely on from the archiving process.
Thanks for the heads-up! However, I'm afraid I don't have the time to constantly test if someone didn't delete the packages we need. If the opam main repository can't be relied upon to provide package versions long-term, we'll have to extend our installation instructions to tell everyone to add the archive repo as well, or we'll copy the dune package to our repo (and miss out on updates, unless there's a way to fully automate that). Maintaining software artifacts is hard enough without having the ground pulled from underneath us by having our dependencies deleted.
I understand you have good reasons for this cleanup, and are operating under conflicting constraints. But as someone who only has a very indirect relation to the OCaml ecosystem (all I want to do is use Rocq and ship Rocq libraries), this approach isn't sustainable for us.
Dear @RalfJung, thanks for your reply. With the package archiving process, we don't demand any "constantly test" something. We are sticking to do at most two archival runs per year. Is this still too much effort? Could there automation be applied (I suspect you have CI systems, and testing with the opam-repository branch as described in the discuss post should be short)?
For "maintaining software artifacts" for papers etc. you're better off to provide a opam switch export --full --freeze output, which doesn't depend on the opam-repository at all.
As discussed several months ago, you can as well submit "virtual" packages (that do not install anything, but depend on the exact versions you use for distribution) to the main opam-repository with your required versions. I'm not sure whether your release engineering has taken up on that (if not, why not - are there specific technical difficulties)?
At the same time, I'm easy. We can as well keep all dune versions in the repository if that makes the life for rocq people easier.
We do have CI, but CI can't know when there's some other repo to test with -- and it would be non-trivial to adjust CI to entirely replace the switch it works in. Also, that is CI in each of our individual repos, of which there's more than a dozen. There's no practical way to test all of them with your repo.
For "maintaining software artifacts" for papers etc. you're better off to provide a opam switch export --full --freeze output, which doesn't depend on the opam-repository at all.
I'll try to remember that for the next artifact, thanks. It won't help for already published artifacts of course, where we assumed that the opam repo is an append-only registry as is the case for most programming language package registries.
As discussed several months ago, you can as well submit "virtual" packages (that do not install anything, but depend on the exact versions you use for distribution) to the main opam-repository with your required versions. I'm not sure whether your release engineering has taken up on that (if not, why not - are there specific technical difficulties)?
And then we'd also have to automate updating that package every time we change our dependencies? That doesn't seem very practical.
We don't have "release engineering" 😂 we are a research group where me and my students spend most of our time doing research but sometimes we have a bit of spare time to do some sysadmin work for our packages. (I don't work on Rocq. I just work with Rocq and maintain a reasonably widely-used Rocq library.)
At the same time, I'm easy. We can as well keep all dune versions in the repository if that makes the life for rocq people easier.
So far, dune is the only package where this has affected us, so that would help. :) Rocq has a few other dependencies though, I guess we'll see if they ever become a problem...
At the same time, I'm easy. We can as well keep all dune versions in the repository if that makes the life for rocq people easier.
So far, dune is the only package where this has affected us, so that would help. :) Rocq has a few other dependencies though, I guess we'll see if they ever become a problem
I removed dune* from that archival proposal. Have a great day.