Allow bumping kernel version without requiring a new SecureDrop release

[legoktm]

Description

It should be possible for us to upgrade the Linux kernel version without requiring a new SecureDrop release.

In https://github.com/freedomofpress/securedrop/issues/6323 we wanted to bump just the kernel for Dirty Pipe, but needed to go through a full SD release cycle for it. Besides the extra unnecessary work, it also causes more churn for SD servers, since they have to upgrade securedrop-app-code packages that are identical except the version number.

[legoktm]

There are two main references to the current kernel version in this git repository.

First the actual linux package we use is named linux-image-5.15.26-grsec-securedrop. This is already separate and "maintained" in the kernel-builder repository.

Then we have a securedrop-grsec package, with a version of 5.15.26+focal, which depends on linux-image-5.15.26-grsec-securedrop,intel-microcode.

We use the actual kernel version in places like testinfra/common/test_grsecurity.py, which verifies the exact kernel version that is installed.

I propose we move the securedrop-grsec packaging to a separate repository, and in places like test_grsecurity.py, we only verify the major+minor version of the installed kernel, so 5.15 in this case. Bumping patch versions can be done independently, requiring just new securedrop-grsec and linux-image-* packages, while a minor or major kernel bump will require a full SD release.

[zenmonkeykstop]

:+1: from me, tis a fine plan.

[conorsch]

with a version of 5.15.26+focal

Ideally, we'd have a version for the metapackage that's independent of the underlying kernel image. But the general outlines of the plan are great, and would indeed free us up a bit. I also support adding the kernel version to the /metadata endpoint so that we can still monitor for successful kernel updates across prod instances. The kernel version is already public information, tied to the app code version, so no change to information shared.

[zenmonkeykstop]

It's possible for instances to downgrade to other grsec kernels, so we don't actually currently know what they're running, we're just inferring it with a reasonable degree of certainty. But overall yeah it's worth the additional information leak. We might also want to consider adding Tor versions to that if we're committing to updating that independently.

[zenmonkeykstop]

(Dang now I'm wondering should we make the hypothetical /version endpoint authenticated, and let instances opt in to us monitoring it by sending us a token or something.)

[legoktm]

@conorsch wrote:

with a version of 5.15.26+focal

Ideally, we'd have a version for the metapackage that's independent of the underlying kernel image. But the general outlines of the plan are great, and would indeed free us up a bit.

I partially agree and disagree. I think the ideal version for the metapackage will be something like 5.15.26-1~focal.

Breaking that down:

• 5.15.26 matches the kernel version. The metapackage is directly tied to a specific kernel version (it depends on e.g. linux-image-5.15.26-grsec-securedrop), so I think reusing that version is fine. We could of course use arbitrary versions, "1.0.0" matches 5.15.26, "1.1.0" matches the next point release we ship (hypothetically 5.15.35), etc. but then that's another mapping we need to keep track of.
• -1 is the Debian package version, if we want to make a change to the packaging without changing the kernel version, we bump this to -2, etc., so the overall version is independent of the kernel. But most of the time it'll probably stay at -1.
• ~focal is the OS version it is built for. Currently we use + nearly everywhere but the convention is to use ~{version}, which I think we should gradually adopt. I can provide a longer rationale for this change if it would be useful, but tbh there's no practical benefit to using tilde outside some very narrow edge cases, it's just following convention, so we could stick with plus too, nbd.

I also support adding the kernel version to the /metadata endpoint so that we can still monitor for successful kernel updates across prod instances. The kernel version is already public information, tied to the app code version, so no change to information shared.

@zenmonkeykstop wrote:

It's possible for instances to downgrade to other grsec kernels, so we don't actually currently know what they're running, we're just inferring it with a reasonable degree of certainty. But overall yeah it's worth the additional information leak. We might also want to consider adding Tor versions to that if we're committing to updating that independently. (Dang now I'm wondering should we make the hypothetical /version endpoint authenticated, and let instances opt in to us monitoring it by sending us a token or something.)

I see no issue with publishing the kernel version and Tor version, they're already public by implication and I think trying to hide them is mostly security by obscurity. I would expect an attacker to just try pulling off an attack against a potentially vulnerable kernel/tor rather than probing the version endpoint first.

If someone is intentionally running old kernels, that seems like a big red flag to me, and a use case we shouldn't support. If we really have to, we could add a config option to just disable this /version endpoint?

Re: having an authenticated endpoint that we have access to, my initial reaction is that I like the current model in which FPF has no special access to SD instances, we only have what's publicly available for everyone and what journalists/admins (privately) tell us about their instance. That's just my initial reaction though, I could be overlooking some argument for it.

[zenmonkeykstop]

Yeah, the authenticated idea is kindof dumb in retrospect. If we do actually monitor for tor/kernel versions and ping instances that are on old versions, it's a net positive, no need to get fancy.

[zenmonkeykstop]

One compromise to consider is to start versioning the securedrop-grsec package with the securedrop application version again, and just incrementing the debian package version if we need to push new kernels - so 5.15.18 would have been pulled in by securedrop-grsec-2.2.0-1, 5.15.26 by securedrop-grsec-2.2.0-2 etc. This doesn't solve the issue of knowing which kernel an instance is running, but it does allow for point releases without the full suite of packages. And we could track the securedrop-grsec version in the metadata api without really exposing any new info.

[legoktm]

Don't we have the same problem then, just in the other direction? If we do a SD release without bumping the kernel, we'd still have to bump the securedrop-grsec package even though there are no kernel changes.

[legoktm]

I uploaded my version of the securedrop-grsec packaging to https://github.com/freedomofpress/securedrop-grsec, it is setup for git-buildpackage and uses git-pbuilder to create a clean build environment rather than docker images. For reproducibility, we'll publish the buildinfo file rather than trying to freeze the build environment. Here's what the debdiff looks like:

user@DD ~/fpf> debdiff securedrop-grsec_5.15.26+focal_amd64.deb securedrop-grsec_5.15.26~focal1_all.deb
[The following lists of changes regard files as different if they have
different names, permissions or owners.]

Files in second .deb but not in first
-------------------------------------
-rw-r--r--  root/root   /usr/share/doc/securedrop-grsec/changelog.gz
-rw-r--r--  root/root   /usr/share/doc/securedrop-grsec/copyright
-rw-r--r--  root/root   DEBIAN/conffiles
-rw-r--r--  root/root   DEBIAN/md5sums

Control files: lines which differ (wdiff format)
------------------------------------------------
Architecture: [-amd64-] {+all+}
Depends: [-linux-image-5.15.26-grsec-securedrop,intel-microcode,paxctld-] {+intel-microde, linux-image-5.15.26-grsec-securedrop, paxctld+}
Description: Metapackage {+with SecureDrop grsecurity-hardened kernel+}
{+ Metapackage+} providing a grsecurity-patched Linux kernel for use with
{+Installed-Size: 23+}
Section: [-admin-]
[-Source: securedrop-grsec-] {+kernel+}
Version: [-5.15.26+focal-] {+5.15.26~focal1+}

Notable changes:

• It's an architecture all package rather than amd64, because nothing in the package itself is dependent upon the amd64 architecture
• Version format changed

All the other changes are pretty insignificant whitespace or that I tweaked the description.

@zenmonkeykstop and I had discussed possibly putting it into the pre-existing kernel-builder repository which I think is technically possible, but it'll be more complicated and messy since there's already a pre-existing Makefile in the repo, plus it's set up to build multiple kernel versions via the same branch, while I'd like to show off using different branches for different distro versions (note that the branch is named ubuntu/focal per DEP-14).

[eloquence]

Stepping back, it may be worth making an explicit choice whether #6355 is a blocker for this issue or not. From my understanding, folks may already be running outdated kernels without us having any visibility into that, so perhaps it would be fine to satisfy this issue before having additional (opt-in or not) tooling for monitoring install state.

[legoktm]

This is technically done now that https://github.com/freedomofpress/kernel-builder/pull/24 has landed, but I will close this issue once the cleanup in https://github.com/freedomofpress/securedrop/pull/6553 is merged.

[legoktm]

The first detached kernel update went out via #6660! All that's left is dropping the obsolete securedrop-grsec metapackage (that PR closes this issue).