theforeman
Add procedures to migrate from RHEL8to9 using backups
Backups can be used to migrate existing Project and Smart Proxy servers from RHEL8 to RHEL9. The backups from Project can be restored using the backup restore as well as clone method. The backups from Smart Proxies can only be restored using the backup restore method. Adding sections to backup and restore/clone.
JIRA link: https://issues.redhat.com/browse/SAT-10790
- [x] I am okay with my commits getting squashed when you merge this PR.
- [X] I am familiar with the contributing guidelines.
Please cherry-pick my commits into:
- [X] Foreman 3.11/Katello 4.13
- [ ] Foreman 3.10/Katello 4.12
- [ ] Foreman 3.9/Katello 4.11 (Satellite 6.15; orcharhino 6.8)
- [ ] Foreman 3.8/Katello 4.10
- [ ] Foreman 3.7/Katello 4.9 (Satellite 6.14)
- [ ] Foreman 3.6/Katello 4.8
- [ ] Foreman 3.5/Katello 4.7 (Satellite 6.13; orcharhino 6.6/6.7)
- We do not accept PRs for Foreman older than 3.5.
The PR preview for d136653fe8b8fa6f0a4540f820feaae5c16a5b7d is available at theforeman-foreman-documentation-preview-pr-3103.surge.sh
The following output files are affected by this PR:
- Upgrading_Project/index-foreman-el.html
- Upgrading_Project/index-katello.html
- Upgrading_Project/index-orcharhino.html
- Upgrading_Project/index-satellite.html
- Upgrading_Project_Disconnected/index-satellite.html
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I only have a two broad comments right now, but before I take a look at this PR in more detail, I'm wondering about the placement of the procedure. @apinnick can you please take a look here? Will users look for a migration procedure in a guide that is called "Upgrading"?
@asteflova No, users would not expect to find this in an Upgrade guide.
This kind of migration is a sub-use case of backup/restore, so it belongs in the Administering guide, which has procedures for migrating databases and for backing up servers.
@apinnick @asteflova so this procedure works as an alternative to the leapp upgrade of the OS which comes under the upgrade guide. Should I move the leapp section as well to Administering guide in that case? For 3.1, where we had EL7 -> EL8 upgrade using leapp, it was placed in the upgrade guide. But I see the point in moving it to Administering guide instead. @evgeni @Lennonka what do you think?
This kind of migration is a sub-use case of backup/restore, so it belongs in the Administering guide, which has procedures for migrating databases and for backing up servers.
But it's a migration with the sole purpose of the upgrade of the underlying OS.
This kind of migration is a sub-use case of backup/restore, so it belongs in the Administering guide, which has procedures for migrating databases and for backing up servers.
But it's a migration with the sole purpose of the upgrade of the underlying OS.
@evgeni In that case, I recommend calling this an upgrade in the title.
It's OK to mention that the upgrade is also a migration, in the introduction. That would eliminate confusion.
So if the Leapp one is called "Upgrading {Project} or {SmartProxy} to {EL} 9 in-place by using Leapp" this one would be "Upgrading {Project} or {SmartProxy} to {EL} 9 by migrating"?
@evgeni @apinnick maybe we can have a chapter for Upgrading from EL 8 to 9, and under it two separate chapters:
- Using LEAPP
- Using Backups
Though technically we are not upgrading using backups, so I'm not very sure about this.
So if the Leapp one is called "Upgrading {Project} or {SmartProxy} to {EL} 9 in-place by using Leapp" this one would be "Upgrading {Project} or {SmartProxy} to {EL} 9 by migrating"?
@evgeni IMO, you don't need "by migrating" unless you have, say, lots of different ways to upgrade {EL}. If that were the case, then you might need "by using migration", to show a distinction between different ways to do the same thing.
In general, short titles are good. :-) They're easier for the user to scan and they minimize the ugly wrap-around effect in the nav menu.
@evgeni @apinnick maybe we can have a chapter for
Upgrading from EL 8 to 9, and under it two separate chapters:1. Using LEAPP 2. Using BackupsThough technically we are not upgrading using backups, so I'm not very sure about this.
Ah, that's a totally different case! You do have two methods for upgrading EL, so it is appropriate to distinguish between the methods. "Using backups" is fine. I might have chosen "by backing up and restoring", but that is longer. Some repos allow "backup/restore". I don't recall whether our guidelines allow that. I'm sure whatever you choose will be fine.
Please ensure that there is a description of the use cases so that users know which upgrade method they should use.
Hello! Out of curiousity i also went through the written steps in this PR to see what things i would stumble across that i can then share. This was done on a machine running the latest foreman-katello as of writing and done on a production machine with snapshots and another form of a backup just in case. We use this machine to provide package updates aswell as to run ansible playbooks from. The host is registered to a Satellite server but get's it foreman and katello packages directly from the internet.
I only have a few minor points related to the steps i took but i'll adress those below as it seems these changes aren't made within this PR!
Let Leapp analyze your system:
The first run will most likely report issues and inhibit the upgrade. Examine the report in the /var/log/leapp/leapp-report.txt file, answer all questions by using leapp answer, and manually resolve other reported problems.
I understood i had to run the leapp preupgrade command, but i would suggest to also mention to run that command for the end user.
Let Leapp create the upgrade environment:
Same as the above, i know i had to run leapp upgrade but the end user may not know that.
This is more a question, but after we are finished with leapp, shouldn't we have to run foreman-installer to make sure foreman-katello is consistent? Would it make sense to refer the user to follow the usual upgrade steps so that we're sure they are on the el9 variant of foreman and katello? And should it be mentioned to update foreman-katello before starting with the leapp upgrade?
Things i stumbled across which may be useful to share
I did apply this fix so i wouldn't run into the grub issue when leaping from el8 to el9: https://github.com/oamg/leapp-repository/pull/1229
I also noticed beforehand that our postgresql version is 13, so we didn't had to do any migration in that regards.
Preupgrade
Only had some minor things to fix, i did had to disable the rhel-8-for-x86_64-highavailability-rpms and codeready-builder-for-rhel-8-x86_64-rpms repo's as i couldn't get the repo's to jump to rhel 9 versions to work. After i upgraded and used the rhel9 activation key i could access these withoud any problems.
Upgrade
During the upgrade i had 2 issues, one wasn't enough room in the /usr folder (easy fix). The other was that i had the following packages that were conflicting with the upgrade:
python39-jinja2-3.1.3-1.2.el8.noarch
python39-lxml-4.9.2-1.el8.x86_64
python39-pycairo-1.20.1-3.el8.x86_64
python39-urllib3-1.26.8-2.el8.noarch
python39-ptyprocess-0.7.0-1.el8.noarch
python39-pytz-2022.2.1-1.el8.noarch
python39-pycparser-2.21-2.el8.noarch
python39-idna-3.3-2.el8.noarch
python39-iniparse-0.4-35.el8.noarch
python39-six-1.16.0-2.el8.noarch
python39-markupsafe-2.1.2-1.el8.x86_64
I removed those and didn't notice they had any relevance to the function of foreman-katello. After that the upgrade was ready and after a reboot i did land on rhel 9.
After upgrade
Foreman wasn't in a working state after the upgrade. I fixed this by running: foreman-installer
I noticed that /etc/yum.repos.d/katello.repo still refered to the el8 repo's and changed that to use the el9 repo's. I also re-activated the machine with a proper rhel9 activation key.
To update the remaining packages from el8 to el9 i did: dnf upgrade --alowerasing . This did remove alot but after running the foreman-installer we could fetch packages withoud any problems.
I noticed because we have EPEL enabled, that we had a problem with rubygem-redis-4.5.1-1.el9 from foreman and rubygem-redis-4.6.0-1.el9 from epel. I disabled the EPEL repo and the issue didn't occure anymore (workaround, but ofcours the foreman rubygem-redis is the correct on to keep).
The only remaining issue with the package manager is this warning: warning: Signature not supported. Hash algorithm SHA1 not available.
Next i looked at selinux and could solve most errors with:
ausearch -c 'pulpcore-worker' --raw | audit2allow -M my-pulpcoreworker
semodule -X 300 -i my-pulpcoreworker.pp
A remaining selinux issue persists SELinux is preventing /usr/bin/python3.9 (and .11) from write access on the file /memfd:libffi (deleted).
I see we have the libffi package installed but i haven't looked if this one is a required dependency for foreman.
When creating the rescue kernel i stumbled across this error: libkmod: kmod_module_parse_depline: ctx=0x555e7955f3e0 path=/lib/modules/5.14.0-427.26.1.el9_4.x86_64//weak-updates/kmod-kvdo/vdo/kvdo.ko error=No such file or directory
But by removing kmod-kvdo and installing it again the issue was resolved.
Overall the upgrade was a success and so far i haven't seen any issue pop-up with selinux on enforcing.
Sorry for the long post 🥔 . But i hope that sharing my experience can be used to finalize/improve the PR.
I have logs related to all the selinux errors i got, what was removed with dnf upgrade --allow-erasing and the preupgrade logs from leapp. I've also kept the logs related to leapp on the system (/var/log/leapp /root/tmp_leapp_py3 /var/lib/leap).
Edit: My bad, it seems i confused the PR to restore from backup with the one of the inplace: https://github.com/theforeman/foreman-documentation/pull/3070/files
Please, rebase. And revise how the current structure works together with the Leapp section.
