check-running-kernel: Compensate for CONFIG_VERSION_SIGNATURE suffix

[amotl]

Dear Bernd and Jan,

while working on recent improvements to the check_running_kernel program at #91, we also found another issue. In order to keep things separated, we've split this amendment off the other patch and want to report about our observations here first.

The problem was that the comparison of running kernel's /proc/version against the on-disk version failed on Ubuntu kernels and derivates, because Ubuntu (and maybe others?) adds another suffix to the on-disk string defined by CONFIG_VERSION_SIGNATURE. At runtime, this string is apparently stripped off and presented through /proc/version_signature instead, so /proc/version does not match the original representation. Sigh.

Based on the findings outlined below, we ask for further guidance. The problem might have to be solved differently than with our ad hoc patch.

With kind regards, Andreas.

---

Introduction

We discovered that on both a bullseye-based PVE/Proxmox machine, and on another vanilla Ubuntu 20.04 machine, both running 5.x Linux kernel versions, there was a suffix added to the on-disk kernel image, which we stripped off using sed in order to satisfy the comparison operation in an ad hoc manner, see patch below.

Observations

We've only been able to spot this on systems running non-vanilla Debian derivates with 5.x Linux kernel versions. On a standard Debian bullseye machine running 5.10.0-12, this is not an issue.

No suffix on vanilla Debian

root@kraftwerk:~$ dd if=/boot/vmlinuz-5.10.0-12-amd64 bs=16913 skip=1 | xzcat | strings | grep "Linux version"
Linux version 5.10.0-12-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.103-1 (2022-03-07)

Empty () suffix on PVE kernel

root@zapato:~$ dd if=/boot/vmlinuz-5.13.19-2-pve bs=17100 skip=1 | zstd -d | strings | grep "Linux version"
Linux version 5.13.19-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) ()

Assigned () suffix on Ubuntu kernel

It's (Ubuntu 5.13.0-30.33~20.04.1-generic 5.13.19), to be precise.

root@next ~$ dd if=/boot/vmlinuz-5.13.0-30-generic bs=17100 skip=1 | zstd -d | strings | grep "Linux version"
Linux version 5.13.0-30-generic (buildd@lcy02-amd64-003) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 (Ubuntu 5.13.0-30.33~20.04.1-generic 5.13.19)

Research

After researching the topic a bit, we discovered that it is apparently an Ubuntu-specific thing, triggered by the CONFIG_VERSION_SIGNATURE compile time option. Indeed, on those systems, there is a synthetic /proc/version_signature file containing this suffix, which does not seem to be present on systems running vanilla Debian.

root@next ~ # cat /proc/version_signature
Ubuntu 5.13.0-27.29~20.04.1-generic 5.13.19

Solution

This patch made things work for us in a quick manner before knowing any details about the background. It might want to be elaborated a bit further now.

Index: pkg-nagios-plugins-contrib/dsa/checks/dsa-check-running-kernel
===================================================================
--- dsa-check-running-kernel.dist	2022-03-12 21:36:28.000000000 +0100
+++ dsa-check-running-kernel	2022-03-12 21:36:32.000000000 +0100
@@ -226,6 +228,9 @@
 	exit $WARNING
 fi
 
+# Adjustment for PVE/Proxmox and Ubuntu kernels: Strip off the last fragment.
+on_disk_version=$(echo "$on_disk_version" | sed -E 's/(.*) \(.*\)$/\1/')
+
 if [ "$(uname -s)" = "Linux" ]; then
 	running_version="`cat /proc/version`"
 	if [ -z "$running_version" ] ; then

References

• https://ubuntu.com/kernel
• https://wiki.ubuntu.com/Kernel/FAQ#Kernel.2FFAQ.2FGeneralVersionRunning.How_can_we_determine_the_version_of_the_running_kernel.3F
• https://stackoverflow.com/questions/70740713/heroku-understanding-what-version-of-linux-kernel-is-used
• https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/[email protected]/

[waja]

I can confirm this issue:

# /tmp/check_running_kernel
WARNING: Running kernel does not match on-disk kernel image: [Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 != Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 (Ubuntu 5.13.0-39.44~20.04.1-generic 5.13.19)]
# cat /proc/version_signature 
Ubuntu 5.13.0-39.44~20.04.1-generic 5.13.19
# cat /proc/version
Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022

Applying the patch would fix it:

# /tmp/check_running_kernel
OK: Running kernel matches on disk image: [Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022]