ocaml-community
Proposal: membership and opam repository
Hi, may I be granted membership with the access level to create and maintain a new repo ocaml-community/opam-repository? The reason for this is that I would like to set up a new community-driven opam package repository to take some pressure off the official ocaml/opam-repository. I envision this as an alternative opam repo that any of us could publish to any time we wanted. This way we would be able to publish without pressuring the official opam-repository maintainers. We can also enforce somewhat more stringent packaging rules, eg we could say that we need a SemVer-like lower-upper bound spec on all dependencies.
Of course, anyone can set up their own opam repository, so why here? My thinking is that this is a nice, somewhat centralized place, to have a community-driven opam repo. There would be a benefit from pooling efforts and reducing the number of different alternative opam repositories if we could convince people to submit their packages here in addition to (or alternative to) the official repo.
Let me know your thoughts. Thanks.
@yawaramin, did you discuss with the opam repository maintainers about "taking some pressure off the official repository"?
It seems to me that it could be more efficient for you to propose your time working as an opam-repository maintainer. It would be also a more straightforward way to achieve "pooling efforts and reducing the number of different alternative opam repositories".
Alternatively, experimenting for a time with your own curated opam repository extension would probably help to settle on a policy and objectives for the repository: your current explanation is quite short on why you think an extended opam repository is necessary. It would still be time to move the repository to ocaml-community once it has a clear raison d'être, a community resource ,and you want to stop working on a maintainer/curator for this repository.
In particular, your propositions of
- enforcing more stringent packaging rule
- allowing anyone to publish any time they want
reads like an antinomy to me. Did you mean that only members of
ocaml-communitywould be able to publish?
Thanks for the feedback @Octachron. True, volunteering to become an opam-repository maintainer could be even more optimal if we look at it purely from the perspective of reducing the number of repos to a minimum. But as you noticed, another goal is a more lax publishing approach where anyone can publish any time. But how to reconcile the contradiction of imposing dependency versioning rules?
How about:
- We require dependency versioning with SemVer-like upper and lower bounds as I said earlier
- We give write access to ocaml-community/opam-repository to anyone who asks for it after validating with them that they agree to abide by the (versioning and other) rules and not abuse the privilege by clobbering other peoples' work. We extend a bit more trust.
Now, anyone with write access will be able to self-publish to the community repo. The official repo has stricter rules and each publish must be approved by a maintainer, which is fine, but the community repo can have different rules.
What do you think?
A few of the rules for the proposed new community repo:
- Packages published here must be OCaml opam packages that use the dune build system
- Packages must specify dependency bounds for most dependencies to a reasonable extent, using SemVer-like bounds eg
"ppxlib" {>= "0.33.0" & < "1.0.0"} - Packages must not be 'hello-world' type projects eg that just print out a message and define some basic bindings like
let foo = 1, they must do something 'useful' (usefulness is broad and open for interpretation but the final judgment rests with the repo owners) - Packages must not be hyper-specialized eg a module which only downloads images of foxes from a specific website, they must be at least somewhat generally applicable
- Packages have names that are distinct from official opam repository packages, unless you are the official opam package's publisher as well. In other words, you can publish your own package in both repos
- Publishers must not overwrite or interfere with others' packages unless specifically requested by the package owner or the community repository owners. We are extending some trust to you, please use it wisely
- Breaking any of the above rules will lead to revocation of access to publish in the community repository.
How to apply for access:
If you are a new publisher, just send a PR with your package similarly to what you would send to the official opam repository. If we merge the PR, we will grant you write access to the repo going forward. Otherwise, we won't :-)
Let me know if you all can think of anything else.
I like this proposal very much.
From my perspective, the biggest issue for the main OPAM repository is packages that are abandoned, either temporarily or permanently. Either way, that disrupts the ecosystem. Any ideas for dealing with that, @yawaramin?
In a package management system that simply uses something like github URIs, a new forked package could swoop in and take over the job of the previous package almost transparently (users will learn which package is up to date from other users). In an environment with a normalized namespace, it's harder for forked packages to find a name for themselves that represents a continuation of the previous package, and there might need to be a way to reuse the same name for different packages or at least an ownership transfer process.
Thanks @bluddy. When packages are abandoned, but someone does want to maintain it, to my understanding the official repo team wants that first the new candidate maintainer must try to establish some communication with the old one and try to get their permission. Sometimes that is successful. Otherwise, I believe after some period of time, opam-repo team would allow the new candidate maintainer to take over publishing the package.
With respect to my proposal, my goal is not to allow 'takeovers' of abandoned official opam-repo packages–I still think the official opam process would be best for that–but to allow people to publish their own packages as quickly as they want to. To that end, I've added a new rule (5) above.
But as you noticed, another goal is a more lax publishing approach where anyone can publish any time.
Anyone can already publish anytime to the opam repository, this is a contradiction. Honestly, I have a hard time to grasp what is the perceived problem that you are trying to solve there.
Moreover, taking in account all the rules that you added, you are not really proposing a laxer opam-repository but a stricter one.
Packages published here must be OCaml opam packages that use the dune build system
Why should a package system care about the build system used? Do you really intend to exclude all the reverse dependencies of cmdliner and uucp? That sounds like a matter of personal taste.
Packages must specify dependency bounds for most dependencies to a reasonable extent, using SemVer-like bounds eg "ppxlib" {>= "0.33.0" & < "1.0.0"}
In other words, you are trying to enforce more stringent dependency versioning rules manually, without a CI to test that this reflect reality? That's sound like a very time intensive curation which is likely to have worse bandwidth issue than the main opam repository.
Anyway, as far as I understand, the current purpose of ocaml-community is to be a home for community project without a core maintainer. Since your project doesn't have a community yet and does have a core maintainer, it stands outside of the current scope of ocaml-community. Should we amend the scope of ocaml-community? I don't know, but so far I am not convinced that your experiment need to be hosted on ocaml-community (and I would still advise you to discuss with opam repository maintainers) before launching your experiment.
Anyone can already publish anytime to the opam repository, this is a contradiction.
There is no contradiction, I am talking about the approved publishers being able to publish by just pushing to the repo directly, without having to go through a pull request and approval for every single publish.
Do you really intend to exclude
Yes, because they are already published on the official repository. They don't need another publish location. They can always just use dune and publish to the community repository, if they want.
That sounds like a matter of personal taste.
That's because a person is proposing this experiment :-)
likely to have worse bandwidth issue
It would be post-facto curation so it wouldn't be a bottleneck (after the initial PR, as I mentioned, we would extend some trust to the approved publishers).
your project doesn't have a community yet and does have a core maintainer, it stands outside of the current scope of ocaml-community.
I realize this, which is why I wanted to start this discussion :-)
I am not convinced that your experiment need to be hosted on ocaml-community
of course, it doesn't need to be hosted here. I can just create a new org 'ocaml-riders' and publish a new opam repo there. I just thought it would be a nice and somewhat easier to find location for the community. If this org's owners are against the idea I will close the ticket :-)
and I would still advise you to discuss with opam repository maintainers
Sure, I can do that. But I don't think the approach I am proposing is a good idea for the official repo, because of its wider audience and the decision to do a repo-wide CI build system.
I like this proposal because currently, the momentum in the official opam repository is to prune packages and focus on a specific set of up-to date packages.
This is designed to relieve the pressure off the opam package repository maintainers. The goal is not to add more hands on deck but to limit the amount of work thus, suggesting to volunteer there instead seems counter-productive.
Likewise, if the official repository is getting pruned, it makes a lot of sense to me to have a more lax repository elsewhere (or several) to allow for packages that are not targeting this more restricted set. This could for many reasons, forks, old versions that some people still want to use etc.
There's already a pretty established pattern of specific opam repository, for instance the opam-cross, the opam repository with packages updated to build with dune (can't find the url right now), the janest opam repository etc.
To me, this proposal is about establishing another one that is focused on a more relaxed, more inclusive repository (inclusive in terms of technical requirement, not saying that the main opam repo is not welcoming here).
Lastly, I welcome this proposal because it fosters the community. I think that every offer to donate time and create space for OCaml developers and maintainers should be supported. We do not need a single, monolithic community, we have space for many different projects and initiatives.
The opam repository is getting pruned of non-installable packages and not-usable versions of existing packages. Thus the premise that the new repository would be a "laxer" repository than the main one seems wrong?
Moreover, this doesn't align with @yawaramin proposal which is already less technically inclusive than the official repository?
(And I don't see why it would be counter-productive to volunteer on the opam repository.)
We do not need a single, monolithic community, we have space for many different projects and initiatives.
I totally agree that we have space for projects and initiatives, and @yawaramin 's idea of launching the repository on a new ocaml-riders organization seems very fine to me. It also would have the advantage of avoiding the potential confusion with the idea of "an opam repository for ocaml-community opam packages".
However, I don't see ocaml-community as a place for such personal experiment, at least without some preliminary discussion on a wider range of community channel. Or maybe turning the question around, I have the feeling that a new initiative may be better served by a new github organization and a new team of people?
Yes, possibly that would be a better approach! Thanks for sharing your thoughts.
the momentum in the official opam repository is to prune packages and focus on a specific set of up-to date packages ... This is designed to relieve the pressure off the opam package repository maintainers. The goal is not to add more hands on deck but to limit the amount of work thus, suggesting to volunteer there instead seems counter-productive
Speaking as an active opam repo maintainer, I would like to reinforce @Octachron's correction here: this is a misunderstanding. The archiving effort is meant to eliminate packages that are not useful, in order to improve the overall operational efficiency of the opam repository system. There is no desire or aim to reduce the amount of new packages being published. The opposite is in fact true. Moreover, the opam maintainers would very much like more volunteers, and it is a grate way to support the community. See https://discuss.ocaml.org/t/call-for-new-opam-repository-maintainers/12041.
I would also like to offer some feedback, speaking purely in an individual capacity, from my own personal perspective.
While I appreciate the general intention here, and @yawaramin consistent concern with the wellbeing of the OCaml community, I am unconvinced about some of the premises here and concerned about some of the plans. Here are my thoughts on points discussed so far:
a new community-driven opam package repository to take some pressure off the official ocaml/opam-repository
opam-repositoryis a community driven opam package repository.- The only way that an alternative repository would "take pressure off of" the
opam-repositoryis if package authors published their packages on this new alternative instead of on the already existing repository, but this also has costs, as now both publishers and users need to face the added complexity of deciding which repository they should be interacting with.
We give write access to ocaml-community/opam-repository to anyone who asks for it after validating with them that they agree to abide by the (versioning and other) rules and not abuse the privilege by clobbering other peoples' work. We extend a bit more trust.
IMO, it would be irresponsible to offer any users a package repository that did not have a meaningful access control policy to ensure that published programs are taken from the sources from which they claim. But how would you do that on top of a mere git repository without curation? If you do have curation (PR reviews, CI checks), then you have introduced actually more pressure on the community, as now we must expend effort on two repositories competing for resources!
This is not an appropriate place to "extend trust" IMO: some sort of access controls and verification should be put in place, or you are setting up a platform that will guarantee the eventual abuse of its users. Imagine the reputational harm to the ecosystem if an alternative package management system, with 0 guardrails against supply chain attacks, actually gained momentum before hurting a bunch of people...
My concerns not withstanding, I do not mean to discourage experimentation. I would just caution against positioning an unproven experiment as a co-equal alternative to a time-tested institution, at least without carefully thinking through second- and nth order effects.
Thanks @shonfeder for sharing your feedback. I have a couple of points here.
opam-repository is a community driven opam package repository.
Sure, I just wanted a way to distinguish between the official opam repo and my proposal, so I am just referring to them as 'official' and 'community'. You can substitute in whatever alternative name you prefer.
package authors published their packages on this new alternative instead of on the already existing repository
That's not my intention. In fact, you can see from my proposed rule (5) that my ideal scenario is that publishers publish on both repos. That actually would take the pressure off the official repo maintainers. Because then we could say to both publishers and users, hey, the official repo requires some human review, which can take slightly longer, but if you also publish on the community repo, it will be out instantly.
I see this as a win-win-win situation: users can get new versions instantly by adding the community repo; publishers can publish instantly; and official opam repo maintainers can relax a bit without having to feel like they are always on the clock.
it would be irresponsible to offer any users a package repository that did not have a meaningful access control policy
I hear you. And I am not terribly attached to the idea of granting full access to the repo to every publisher. An alternative plan could be to require a PR for the first package submission from any publisher; the PR would be vetted and merged just like the official repo. After that, we can have a cron job in GitHub Actions that runs every few hours and checks for new tagged versions of already-published packages, on a rotating basis, and publishes the new versions. We can even use the x-maintenance-intent field to delete the old versions of the package if desired.
So each publisher would effectively be able to publish only their own packages, and not mess around with any others. I think this is a pretty good compromise between safety guardrails and publish speed. Would love to hear your thoughts.
I (being an opam-repository maintainer but voicing my own personal opinion) think that this additional opam-repository is a good idea. I'm not convinced it'll take pressure off of the ocaml/opam-repository maintenance, at least not in the short term. But that's ok.
I think an additional opam-repository could be a good place to experiment with different rules, policies, practices, tooling, etc. E.g.,
- semantice versioning (as mentioned earlier in the thread): What does it look like in OCaml? How to enforce it via tooling? E.g., can we make a lint check that diffs two consecutive version of a package and checks that the versioning is indeed semantic. Or is that too complicated and we just need guidelines for package authors?
- versioned external dependencies: How to specify needing
rustedition 2018 or python >= 3.11 or some such? The conf packages often ommit version information for the underlying system packages so it's often hard to do. - namespacing: Can we work namespacing into opam packaging? Should we?
These and probably other questions are hard to answer on ocaml/opam-repository because it's hard to experiment on a working live system. But maybe an alternative repository would be a good place for someone to start publishing namespaced semantically-versionned packages…
Another potential of this alternative repository is to advertise the fact that it is possible (and even not too hard) to host your own package repository. This could be useful for self-hosting for packages that don't necessarily need to be broadcast to the whole ocaml community but still offer opam-backed installation.
In the context of the archive process for opam in this issue I had made a comment with a proposal for a repo of repos. Back then it didn't generate any comment, possibly it was out of scope. Maybe this issue is a better fit to have a discussion about it?
Let me retry re-posting here:
What I was thinking of was to create a new repository for opam, which would list custom opam repositories that various users have (examples here, here, etc).
This "meta" repository would be a registry of link towards other repos, so in essence a map from a name, to a git URL. This is an "opam-repos-registry" rather than a "opam-packages-registry".
I am imagining that the process of adding a new entry to this repo could be similar to that of the existing main opam repository. Users would add an entry with a link to their repo, reserving a part of a new global namespace composed of the repo name (GitHub username, company name, etc.) and the name of individual packages defined there.
The barrier for inclusion could include a linting step for the added configuration files, with some basic verification that the URL given is indeed pointing to a valid opam-repository. We could certainly draw inspiration from the current process in the main opam-repository for deciding what other considerations are important before merging PRs.
By design, the opam maintainers would defer to the maintainer of each repo for defining the policy on maintenance, lifetime, and quality of packages listed in their custom repo.
This is not a fully fleshed-out proposal, it's an idea that we could implement now. It could provide useful data from the community and potentially be part of a larger, incremental solution. If the repo gains traction, we could consider modifying some tools to gradually expose its data. For example including the packages there into tools like sherlodoc, etc.
In a world where you'll soon be able to specify package dependencies directly with dune without requiring packages to be listed centrally, this could also offer a way for packages living outside the main repo to be discovered. Additionally, the unique repo names defined in the repos-registry would provide a canonical reference for each package.
I'm open to discussing this further if some find this idea appealing. I believe it has several beneficial properties, including the fact that it doesn't require immediate changes to existing tools (opam handles custom opam-repositories beautifully), and it relates to the topics discussed here.
Adding now more contents as it relates to this issue:
Personally, I'd be a bit uncomfortable adding to my opam switch the entire set of packages coming from a single repo of "unstable packages" with little to no vetting process.
However, I'd be comfortable getting familiar with a few chosen custom repos of interest, and add these specifically, each with their own vetting process.
If such repo existed, I would submit an entry with my username, say, pointing to my custom repo:
{ name = "mbarbin"
; url = "https://github.com/mbarbin/opam-repository.git"
...
}
Once the place and format where this information lives is stable and in place, you can imagine submitting PRs to commonly used tools such as opam and dune, that would make use of the information of where this information lives, and its format. For example, if opam is made aware of it, you can imagine that:
Instead of running:
opam repo add mbarbin https://github.com/mbarbin/opam-repository.git
Users would be able to instruct:
opam repo add mbarbin --community
Or perhaps even directly:
opam install mbarbin/dunolint
Would would under the hood perform the already supported steps of:
- Adding the repo with the url resolution with the community owned repo-of-repos
- Install the package from there
Currently I am suggesting using such custom-opam repo to distribute projects, such as in this documentation page (dunolint install guide).
@yawaramin I was notified this issue was closed. I am curious about what was the resolution in the end. Thank you!
Hi @mbarbin no real resolution. The main suggestion was that I could try to make my own opam repository with the rules I proposed and see if it works out.
Got it! Welcome to the club of folks having their own opam repository! Happy to chat whenever. Thank you for letting me know.