The "stable" vs. "next" EPEL 10 repository variants in Copr?

[praiskup]

There's the Mock issue https://github.com/rpm-software-management/mock/issues/1427 that discusses how to work with the future EPEL minor versions. We should consider the approach for Fedora Copr.

I think I'm sure we don't want to have epel-10.0-*, 10.1, ... variants in Copr. We probably want to have two epel variants, one for the RHEL users (building against the latest released minor) and one for CentOS Stream users (building against the next minor, previously called "epel next").

I don't think it matters much how we name the build targets (the chroot selection form self-documents the options). The problem I think is what happens after dnf copr enable <user/project> on C10S and RHEL 10 (and other EL variants).

[praiskup]

We could have rhel+epel-10-x86_64 (stable) chroot in Copr, then centos-stream+epel-10-x86_64 (next), but then the question is what to do with epel-10-x86_64 chroot. Also, the dnf5 copr enable implements fallback mechanism, I'm curious if C10S or RHEL users want to do any fallbacks to the counterpart repo or not.

[FrostyX]

I think I'm sure we don't want to have epel-10.0-*, 10.1, ... variants in Copr.

Yeah, that sounds horrible for so many different reasons

I don't think it matters much how we name the build targets

Agreed. Also, we have a feature for creating aliases, so renaming them if needed, shouldn't be hard

We could have rhel+epel-10-x86_64 (stable) chroot in Copr, then centos-stream+epel-10-x86_64 (next)

Makes sense to me

but then the question is what to do with epel-10-x86_64 chroot

I would probably do the same thing that you did in Mock and not have epel-10-x86_64 at all.

[praiskup]

I would probably do the same thing that you did in Mock and not have epel-10-x86_64 at all.

Even if In Fedora Koji still has epel-10 (has some default meaning)? E.g., koji mock-config --target epel10 --arch x86_64 which is C10S + EPEL "next". I think I still tend to have as close defaults as possible to normal Fedora.

[carlwgeorge]

The core principle of the EPEL 10 changes is to integrate with RHEL minor versions, including CentOS Stream as the leading minor version. Ignoring minor versions has hurt EPEL for a long time. I went into this in detail in this conference talk from earlier this year. We should only gloss over the minor versions when it's absolutely necessary. Here is where we've done it so far:

• The leading branch of epel10: We don't want maintainers to get rejected when they request the branch name they expect. We learned with EPEL Next that many maintainers won't read announcements or documentation, and we should make it easy to accidentally do the correct thing. It's also fairly easy to map this into 10.0 builds, and in the future switch it to 10.1 builds and so on. This also carries through to the epel10-candidate target in koji (which currently uses the epel10.0-build tag) so fedpkg build can guess the right target from the branch name, and the epel-10-*.cfg mock configs for fedpkg mockbuild.
• The leading repo directory as epel/10: CentOS Stream doesn't have a minor version, or rather it doesn't know what minor version repo to request. So on the mirrors we sync out the leading minor version compose to a repo directory without a minor version. In the future we'll have the trailing minor versions in repo directories like epel/10.0. RHEL does know it's minor version, so it can request the specific minor version repo that matches itself.

I believe the simplest thing to do is the match the core structure of EPEL 10 in other parts of the ecosystem such as mock and copr. That means epel-10-*.cfg + epel-10.0-*.cfg + ... mock configs and epel-10 + epel-10.0 + ... copr targets. I know the argument against it is maintaining more targets over time, but I still think in the long term this will be less work than inventing ways to gloss over the minor versions. It is also no more work than Fedora targets that change at about the same rate (every six months). My team plans to build in this work in our EPEL releng SOP, so we can send the pull requests to add the new minor versions over time to help share the maintenance burden. mock-core-configs already has to release at least every six months, and the same release that adds the next Fedora version can also add the next EPEL 10 minor version.

If we go with my recommendation to integrate with minor versions, then we can use the same approach to dnf copr enable that we use for the main repo metalinks in epel-release:

metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever_major${releasever_minor:+.$releasever_minor}&arch=$basearch

The variable expansion there means that C10 users request an epel-10 repo, and RHEL10 users request an epel-10.0 repo. This matches the directory structure we have on the mirrors.

I would probably do the same thing that you did in Mock and not have epel-10-x86_64 at all.

I'm not sure what you mean here, we added epel-10-* mock configs in https://github.com/rpm-software-management/mock/pull/1484.

[praiskup]

My team plans to build in this work in our EPEL releng SOP, so we can send the pull requests to add the new minor versions over time to help share the maintenance burden.

Great! We can give you access to maintaining our Copr instances (we have 4 instances right now), because help here will be absolutely needed. But note that maintaining the minor chroot versions is proven to be painful in Copr, and I'm not convinced you want to go this way. Namely: Copr backend doesn't inherit 10.0 builds into 10.1 when it gets created. Users don't get new chroots enabled by default in their projects. While not impossible to implement (Copr upstream), I'm still not convinced it is worth it.

Is it really known fact that we will always build only minor -1, but not minor -3 (like with RHEL EUS)? Then this is really asking for "stable" vs. "devel" targets.

mock-core-configs already has to release at least every six months

We try to avoid the pressure caused by mock releases as much as possible, though. We know how painful is the Fedora branching event, so the process is semi-automatized, and we made it so it can be done "a month in advance". Otherwise, you have to count with delays, e.g., fixing the main branch so it builds (after previously merged PRs), that test still pass, creating Fedora updates, and then waiting till users give us karma in Bodhi (and respins). It simply doesn't scale well.

Then what I really dislike is that mock-core-configs (+ Copr) is another source of truth here - while Bodhi knows what is the latest EPEL 10 minor version, and we have to solve this on Mirrors (SOP) -- we probably have to touch fedora-distro-aliases, Mock, and also Copr.

[carlwgeorge]

Namely: Copr backend doesn't inherit 10.0 builds into 10.1 when it gets created.

That is an aspect I hadn't considered yet. One of the key parts of the plan is that koji builds get inherited into future minor versions. We plan to accomplish this with commands like koji clone-tag --all --latest-only epel10.0 epel10.1. A maintainer could create a build for epel10.0, and if a newer build never happens then that same build will later be tagged for epel10.1, epel10.2, and so on. Is there a way to accomplish something similar in Copr?

Users don't get new chroots enabled by default in their projects. While not impossible to implement (Copr upstream), I'm still not convinced it is worth it.

Since we have the option of "Follow Fedora branching", I think an option for "Follow EPEL branching" would make sense.

Is it really known fact that we will always build only minor -1, but not minor -3 (like with RHEL EUS)?

EUS is explicitly out of scope for the initial rollout of EPEL 10, but it is something we have an eye towards considering in the future. Most of the time there will only be two active EPEL 10 releases. However, because of the gap between a RHEL minor version being branched and being released, there will be short periods where we will have three active releases. For example, once RHEL 10.1 branches from CS 10, CS 10 will be eligible to receive 10.2 content, but public RHEL will still be on 10.0. At that point we'll have EPEL 10.0, 10.1, and 10.2. After the RHEL 10.1 release, we'll retire EPEL 10.0 and just have EPEL 10.1 and 10.2.

[praiskup]

Sorry for the delay, got busy elsewhere and I needed to absorb this a bit.

One of the key parts of the plan is that koji builds get inherited into future minor versions.

This makes sense.

Is there a way to accomplish something similar in Copr?

This is to some extent possible, but I'm still convinced it doesn't make sense to waste so big amount of development and maintenance efforts for something that can be just covered by floating targets like epel-10 and epel-10-stable, doing the move in one place. Even this needs to be covered in dnf4 copr enable and dnf5 copr enable (we need to teach the DNF plugins to enable appropriate chroots according to what we think that particular user wants).

Since we have the option of "Follow Fedora branching", I think an option for "Follow EPEL branching" would make sense.

This is another non-trivial contribution that would have to be done, feel free to start here: https://github.com/fedora-copr/copr/issues/1026 (then we'll need a maintenance Copr command for /bin/copr-frontend branch-epel (corner cases like epel-9 and smaller need to be covered, etc.). Then we'll need a help with doing this periodically.