fedora-coreos-config
fedora-coreos-config copied to clipboard
coreos
write validation to let us know when packages in `next-devel` are older than `testing-devel`
The idea here is that we never want someone upgrading to have package downgrades (unless we explicitly decided to do that). In the period of time where next is on the N+1 Fedora if packages in that stream sort older (NVR) than what is in the testing streams things are setting up such that packages will "downgrade" at some point as part of an update. This "downgrades" are common in the lead up to the release of the next Fedora major version because there are periods of "freeze" for N+1, but N continues rolling.
One way to get in front of this is to make sure packages in next-devel always NVR sort newer than packages in testing-devel.
Let's write a bot that will let us know when this happens.
Today I use the following steps to determine which packages are downgraded:
podman run -i --rm registry.fedoraproject.org/fedora:36 <<EOF
dnf install -y fedora-repos-ostree ostree rpm-ostree
mkdir /tmp/repo
cd /tmp/repo
ostree init --mode=archive --repo=.
cat /etc/ostree/remotes.d/fedora-compose.conf >> ./config
ostree pull --commit-metadata-only --depth=0 fedora-compose:fedora/x86_64/coreos/testing-devel
ostree pull --commit-metadata-only --depth=0 fedora-compose:fedora/x86_64/coreos/next-devel
rpm-ostree --repo=. db diff fedora-compose:fedora/x86_64/coreos/testing-devel fedora-compose:fedora/x86_64/coreos/next-devel
EOF
For example, as of today the output includes:
Downgraded:
ethtool 2:5.17-1.fc35 -> 2:5.16-2.fc36
runc 2:1.1.1-1.fc35 -> 2:1.1.0-2.fc36
squashfs-tools 4.5.1-1.fc35 -> 4.5-18.20220221gitbc0c097.fc36
vim-data 2:8.2.4701-1.fc35 -> 2:8.2.4621-1.fc36
vim-minimal 2:8.2.4701-1.fc35 -> 2:8.2.4621-1.fc36
So I'll go through and find updates for each of these and then add them as overrides to next-devel. For example for vim we can find the latest packages NVR in bodhi via the bodhi CLI (dnf install bodhi):
$ bodhi updates query --releases f36 --packages vim
*vim-8.2.4701-1.fc36 rpm testing 2022-04-07 (6)
*vim-8.2.4621-1.fc36 rpm stable 2022-03-29 (15)
vim-8.2.4579-1.fc36 rpm unpushed 2022-03-17 (26)
vim-8.2.4529-1.fc36 rpm unpushed 2022-03-09 (34)
vim-8.2.4485-1.fc36 rpm unpushed 2022-03-01 (42)
*vim-8.2.4460-1.fc36 rpm unpushed 2022-02-24 (47)
vim-8.2.4428-1.fc36 rpm stable 2022-02-21 (51)
vim-8.2.4386-1.fc36 rpm stable 2022-02-15 (57)
vim-8.2.4314-1.fc36 rpm stable 2022-02-10 (62)
vim-8.2.4232-1.fc36 rpm stable 2022-01-27 (75)
vim-8.2.4198-1.fc36 rpm stable 2022-01-24 (79)
vim-8.2.4068-1.fc36 rpm stable 2022-01-12 (90)
vim-8.2.4051-1.fc36 rpm stable 2022-01-10 (93)
vim-8.2.4006-1.fc36 rpm stable 2022-01-05 (98)
vim-8.2.3811-1.fc36 rpm stable 2021-12-15 (119)
vim-8.2.3755-1.fc36 rpm stable 2021-12-07 (127)
vim-8.2.3717-1.fc36 rpm stable 2021-12-02 (132)
vim-8.2.3642-2.fc36 rpm stable 2021-11-26 (138)
vim-8.2.3642-1.fc36 rpm stable 2021-11-22 (142)
vim-8.2.3582-1.fc36 rpm stable 2021-11-08 (156)
And we can find the update URL from that:
$ bodhi updates query --releases f36 --builds vim-8.2.4701-1.fc36
================================================================================
vim-8.2.4701-1.fc36
================================================================================
Update ID: FEDORA-2022-44f5b2df35
Content Type: rpm
Release: Fedora 36
Status: testing
Type: security
Severity: urgent
Karma: 5
Autokarma: True [-3, 3]
Autotime: True
CI Status: All required tests passed
All required tests passed
Request: stable
Bugs: 2070437 - CVE-2022-1160 vim: heap buffer overflow in
: get_one_sourceline
: 2070438 - CVE-2022-1160 vim: heap buffer overflow in
: get_one_sourceline [fedora-all]
: 2073013 - CVE-2022-1154 vim: use after free in utf_ptr2char
: 2073014 - CVE-2022-1154 vim: use after free in utf_ptr2char
: [fedora-all]
Notes: The newest upstream commit
: Security fix for CVE-2022-1160
: Security fix for CVE-2022-1154
Submitter: zdohnal
Submitted: 2022-04-07 09:30:32
Comments: bodhi - 2022-04-07 09:30:33 (karma 0)
This update has been submitted for testing by zdohnal.
bodhi - 2022-04-07 09:30:34 (karma 0)
This update's test gating status has been changed to 'waiting'.
bodhi - 2022-04-07 10:04:44 (karma 0)
zdohnal edited this update.
bodhi - 2022-04-07 12:37:28 (karma 0)
This update's test gating status has been changed to 'failed'.
bodhi - 2022-04-07 13:33:15 (karma 0)
zdohnal edited this update.
bodhi - 2022-04-07 13:33:17 (karma 0)
This update's test gating status has been changed to 'waiting'.
bodhi - 2022-04-07 15:34:03 (karma 0)
This update's test gating status has been changed to 'failed'.
bodhi - 2022-04-07 18:02:14 (karma 0)
This update has been pushed to testing.
bodhi - 2022-04-07 18:31:23 (karma 0)
This update's test gating status has been changed to 'passed'.
nixuser - 2022-04-07 23:20:39 (karma 1)
This is working fine.
geraldosimiao - 2022-04-08 13:54:33 (karma 1)
bodhi - 2022-04-08 13:56:12 (karma 0)
This update can be pushed to stable now if the maintainer wishes
atim - 2022-04-08 14:28:40 (karma 1)
bodhi - 2022-04-08 14:28:41 (karma 0)
This update has been submitted for stable by bodhi.
andilinux - 2022-04-09 21:14:41 (karma 1)
works good
lruzicka - 2022-04-12 12:36:37 (karma 1)
Works brilliantly.
https://bodhi.fedoraproject.org/updates/FEDORA-2022-44f5b2df35
1 updates found (1 shown)
Now we just need to run it through overrides.py to update the overrides manifest:
./ci/overrides.py fast-track -r https://github.com/coreos/fedora-coreos-tracker/issues/1101#issuecomment-1085987923 https://bodhi.fedoraproject.org/updates/FEDORA-2022-44f5b2df35
NOTE: you can pass more than one bodhi URL to overrides.py so you can collect all the bodhi URLs first before running overrides.py.
If you can reduce that process entirely to code, it would make sense to add a periodic GitHub Actions workflow that submits PRs, based on add-override.yml.
Exactly. This was the start of that. I wanted to get my steps down and then someone (or maybe me) can pick it up and push it forward. This was step 1.
This might not even be a bot, it could be a check added to the pipeline which warns if an older package is detected.
This could even be a test that is in the config repo. (backlog grooming discussion)
I'm attempting to write some automation for this but I've ran into a couple of scenarios that need to be addressed:
-
overrides.pyfails with some bodhi updates due to multi-arch components so it will probably need to be updated to handle that case. - sometimes bodhi updates are not available for the F(n+1) release because the package maintainer hasn't submitted one yet. This could be handled as a warning output in automation, but eventually we'll need a human-in-the-loop (HITL) to resolve that issue.
What other scenarios could be a challenge?
What I've been doing right now is a mixture of automation and HITL. I'm running a script that performs the steps outlined by Dusty to produce a list of packages that are showing as downgraded along with their bodhi updates for F(n+1). I then have to manually update the manifest lock files or call overrides.py manually.
Nice. Thanks for working on this. Yes if we could update overrides.py to handle packages that are specific to particular architectures that would cause us much less pain not only for this but normal overrides when we do them.
Warnings sound good to me when there are cases that need extra attention.
One other case we migth want to flag is when the update we are fast-tracking isn't currently in the testing->stable state. We'd prefer to only fast-track packages that are in that state to know we're not shipping something that won't be in the rest of fedora eventually.