[BUG] AWS AlmaLinux 8 -> 9 upgrade failed

[ddaddy]

Describe the bug No blockers where reported so I started the process. After stage 3, I no longer could access the server via SSH. Using the AWS console I tried to resume the script, but was met with:

error while loading shared libraries: libnsl.so.2: cannot open shared object file: No such file or directory

The server had no network access either. Using ChatGPT to help, I got network access back using these commands:

# Stop legacy network service if present (avoid conflicts)
systemctl disable --now network 2>/dev/null || true

# Unmask + start NetworkManager
systemctl unmask NetworkManager
systemctl enable --now NetworkManager

# Confirm NM is running
nmcli -t -f RUNNING general
nmcli device status

# Create a simple DHCP profile for eth0 and bring it up
nmcli con add type ethernet ifname eth0 con-name primary ipv4.method auto ipv6.method ignore || true
nmcli con up primary

# You should now have an IP and a default route:
ip -4 a
ip r

Apparently AlmaLinux 9 ships with libnsl.so.3 not libnsl.so.2 so I used these commands to get scripts to be able to run again:

# make sure the dir exists
mkdir -p /usr/local/cpanel/3rdparty/lib64

# point the old soname to the EL9 lib temporarily
ln -sf /usr/lib64/libnsl.so.3  /usr/local/cpanel/3rdparty/lib64/libnsl.so.2

# (you already have this one, but ensure it's visible to cPanel Perl)
ln -sf /usr/lib64/libcrypt.so.1 /usr/local/cpanel/3rdparty/lib64/libcrypt.so.1

# test cPanel’s Perl
LD_LIBRARY_PATH=/usr/local/cpanel/3rdparty/lib64:/usr/lib64 \
/usr/local/cpanel/3rdparty/bin/perl -V | head

I then attempted to repair the panel packages with:

/scripts/check_cpanel_pkgs --fix
systemctl restart cpanel || true

which seemed to do well but ended with:

[2025-09-03 13:55:26 +0200]     libidn is needed by cpanel-perl-536-net-libidn-0.12-1.cp108~el9.x86_64
[2025-09-03 13:55:26 +0200]     libidn is needed by cpanel-exim-4.98.2-1.cp118~el9.x86_64
[2025-09-03 13:55:26 +0200] E The following possible errors were detected while installing RPMs:
[2025-09-03 13:55:26 +0200] E Failed dependencies
[2025-09-03 13:55:26 +0200]   The Administrator will be notified to review this output when this script completes
[2025-09-03 13:55:26 +0200]   Hooks system enabled.
[2025-09-03 13:55:26 +0200]   Checking for and running RPM::Versions 'post' hooks for any Packages about to be installed
[2025-09-03 13:55:26 +0200]   All required 'post' hooks have been run
[2025-09-03 13:55:26 +0200]   Restoring service monitoring.
/scripts/check_cpanel_pkgs: line 21: syntax error near unexpected token `('
/scripts/check_cpanel_pkgs: line 21: `use Cpanel::Usage               ();'
[root@myserver ~]# systemctl restart cpanel || true
Job for cpanel.service failed because the control process exited with error code.
See "systemctl status cpanel.service" and "journalctl -xeu cpanel.service" for details.
[root@myserver ~]# systemctl status cpanel.service
× cpanel.service - cPanel services
     Loaded: loaded (/etc/systemd/system/cpanel.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2025-09-03 13:56:15 CEST; 12s ago
    Process: 1855 ExecStart=/usr/local/cpanel/scripts/restartsrv_cpsrvd --no-verbose --notconfigured-ok --systemd-service=cpanel (code=exited, status=127)
   Main PID: 1855 (code=exited, status=127)
        CPU: 1ms

Sep 03 13:56:15 myserver.vatmtd.com systemd[1]: Starting cPanel services...
Sep 03 13:56:15 myserver.vatmtd.com restartsrv_cpsrvd[1855]: /usr/local/cpanel/scripts/restartsrv_cpsrvd: error while loading shared libraries: libnsl.so.2: cannot open shared object file: No such file or directory
Sep 03 13:56:15 myserver.vatmtd.com systemd[1]: cpanel.service: Main process exited, code=exited, status=127/n/a
Sep 03 13:56:15 myserver.vatmtd.com systemd[1]: cpanel.service: Failed with result 'exit-code'.
Sep 03 13:56:15 myserver.vatmtd.com systemd[1]: Failed to start cPanel services.
[root@myserver ~]# /scripts/elevate-cpanel --status
Running stage 4 / 5

I've then attempted to continue the elevate process:

/scripts/elevate-cpanel --continue

which although it ran and completed after another reboot, it still gives a lot of them libnsl.so.2 errors, so I don't know if it has skipped important stuff.