fedora-copr
option to keep old builds around?
Original issue: https://pagure.io/copr/copr/issue/1365 Opened: 2020-04-30 18:15:37 Opened by: crobinso
I can't seem to find a clear authoritative answer so I'm asking here: is there any way to have a COPR repo distribute multiple versions of the same package? So, libvirt-6.0.0-1 AND libvirt-6.0.0-2. Currently it seems that when a new build is submitted, the previous one is no longer distributed.
If this option isn't available, is it a valid RFE or has this been previously rejected? Links are fine.
My usecase: some projects use virt-preview copr in their CI (ovirt and kubevirt). virt-preview rebuilds rawhide qemu and libvirt and friends but against stable Fedora releases. Once a new libvirt version is released, it kicks out the old ones, despite those upstream projects not necessarily wanting to move to a new version yet. If we could keep old versions around and available in the repo for some period of time it would simplify some upstream issues quite a bit.
praiskup commented at 2020-05-04 05:09:23:
Thanks for the report.
Currently it seems that when a new build is submitted, the previous one is no longer distributed.
We don't delete builds which are not older than 14 days. But yes, because storage is the most painful constraint, we can not simply host all the builds ever done.
There is (copr admin only) option to mark particular copr as "persistent", meaning that no builds can be removed at all. Which is quite unpractical for you I'd guess, it is supposed to mimic Koji's behavior with production builds (they are never removed).
If we could keep old versions around and available in the repo for some period of time it would simplify some upstream issues quite a bit.
The disk space will stay the number one problem for us and we can not simply allow anyone to keep every single build done in Copr.
So my questions would be:
-
Can't you simply have multiple repositories for multiple versions of libvirt?
-
Can't you live with multiple package names?
-
Are you able to tell that at most N parallel versions of the same package name should be kept? Than we could be valid RFE for prunerepo, and we could add config option for it.
-
Would you be able to tell what particular builds are important to keep, explicitly and post-build time? Meaning that if we allowed users to mark at most N particular builds as "persistent", it would be enough for you? This one would be pretty hard RFE to implement, though.
Team, any more ideas?
crobinso commented at 2020-05-06 20:30:32:
Thanks for the consideration.
Can't you simply have multiple repositories for multiple versions of libvirt?
Yes this is probably the solution I will take, one repo per libvirt build and one repo per qemu build. It's a bit noisy but I think it will work, just requires more setup on my side. (I was planning to do this anyways because even if you guys approve the feature it's not going to appear overnight)
Can't you live with multiple package names?
I'm not sure what you mean here
Are you able to tell that at most N parallel versions of the same package name should be kept? Than we could be valid RFE for prunerepo, and we could add config option for it.
Maybe, but it would be a guess on our side. For the use case I'm thinking of I don't think timeframe is necessarily predictable. Unless you can set the timeframe to 'forever', which would map to keeping builds around as long as they are still valid for an existing buildroot.
Would you be able to tell what particular builds are important to keep, explicitly and post-build time? Meaning that if we allowed users to mark at most N particular builds as "persistent", it would be enough for you? This one would be pretty hard RFE to implement, though.
That would probably work a bit better than just time based, but yeah sounds like a lot of work on your side.
praiskup commented at 2020-05-06 21:10:33:
Can't you live with multiple package names?
I'm not sure what you mean here
Packages like libvirt600|libvirt601. If the name is different, multiple versions
will be kept in the repo.
crobinso commented at 2020-05-07 00:46:53:
Packages like libvirt600|libvirt601. If the name is different, multiple versions will be kept in the repo.
At least for my needs, anything that requires changing the spec file is not worth the hassle. The one-project-per-version approach is doable though
This is needed for example when you have a package foo-1 that requires bar-1. Then you want to update foo to foo-2 but it requires bar-2. So you need to build bar-2 first, which causes bar-1 to be removed. But foo-2 is quite a bit of work to get working so it takes longer to get built in the copr. So while that is still a WIP, foo-1 no longer works because bar-1 was removed due to bar-2.
It seems there needs to be some kind of dependency check on the pruning of old versions that removing them will not break any other package in the copr. This is sane because then bar-1 remains, even after bar-2 is built because foo-1 is still the latest foo in the repo and there is a dependency relationship on bar-1 from foo-1 telling you not to prune bar-1 yet.
Once foo-2 is successfully built, foo-1 can be pruned because there is nothing depending on it and once it has been pruned bar-1 can be pruned, thus satisfying your requirement to conserve disk space as much as possible (without breaking anything -- surely not breaking anything must be a part of the disk space conservation requirement).
Does the above solution satisfy your requirement in this issue @crobinso? If not and you still want multiple old builds kept around regardless of dependency breakage or not, I can open a new issue for my suggestion above.
@brianjmurrell thanks for following up! that sounds useful but I don't think it would help my usecases. really I'd like an option to just keep every build around, or at least X number up to 10 or something
Consider the case where you use copr to ship some app that's not in fedora, FOOBAR-1. You push FOOBAR-2 to copr, copr deletes FOOBAR-1. Sometime later a user dnf updates FOOBAR-1 to FOOBAR-2, but hits a massive regression like app crashing on startup. There's no dnf downgrade option to get back to working FOOBAR-1. User needs to wait for a new build to be pushed to the copr.
In my work in virt land where we use copr for a variety of reasons, this is a concern I have to keep in mind all the time. At times I have spun up entirely separate coprs to preserve old versions of packages so people have a viable downgrade path if new versions are broken for their usecases.
This behavior was also considered when the virt team decided to use centos virt SIG for distrbuting not-yet-upstreamed kernel TDX packages for partner testing. See for example how this directory has many kernel builds in it. This is useful for identifying regressions: https://mirror.stream.centos.org/SIGs/9-stream/virt/x86_64/tdx-devel/Packages/k/
It's an ongoing concern for debuggability of packages in the widely used virt-preview repo too: https://fedoraproject.org/wiki/Virtualization_Preview_Repository
I understand storage is a concern, and I've largely accepted this is how copr works and learned to live with it. But if yall added an option I'd be absolutely thrilled :)
@crobinso Indeed, dnf downgradeability is very nice to have.
I wonder if time is a suitable factor in keeping old builds vs. simply a count. I.e. being able to specify to keep old builds for, say, 3 months. That would give people a 3 month window of dnf downgradeability.
@brianjmurrell it would be better than nothing but still not ideal IMO. There are specific cases I can think of where we were providing packages for kubevirt in the past where this would have been a problem.
At least for copr's targeting stable Fedora releases, there's a natural timeout when distro goes EOL after 13 months, and chroots are deleted.