Vulnerabilities (CVEs) for kernel images on Ubuntu

[noahtalerman]

trafficstars

Goal

User story
As a vulnerability management engineer,
I want to see vulnerabilities (CVEs) for kernel images on Ubuntu (DEB package)
so that I can see which vulnerabilities affect my Linux hosts.

Context

• Product designer: @noahtalerman

Changes

Product

• [ ] Vulnerability processing changes: Surface CVEs for kernel images based on their x.y.x-* version. For example if there's a CVE for version 5.17.15, surface this CVE if a host has a kernel image w/ version 5.17.15 or 5.17.15-foo.bar~hotdog.
• [ ] UI changes: No UI changes. Kernel images on Ubuntu now show CVEs in the UI.
• [ ] REST API changes: No API changes. Kernel images on Ubuntu now show CVEs in the relevant Fleet API endpoints.
• [ ] Permissions changes: No permissions changes.
• [ ] Outdated documentation changes: Update the "except for kernel vulnerabilities" in the Coverage section on the Vulnerability processing page:
• [ ] Changes to paid features or tiers: CVEs are available in Fleet Free and Fleet Premium. EPSS, CVSS, CISA KEV, publish date, and description are Premium only.

Engineering

• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[noahtalerman]

Today, Fleet collects deb packages. When I filtered this Linux host in dogfood using a 6.2.0 search query, I got these results:

I'm not the expert but my guess is that linux-image is the Linux kernel while the other packages are installed when the kernel is installed.

None of them have vulnerabilities.

Let's say kernel version 6.2.0 has CVE-9999-9999. Would we expect to see CVE-9999-9999 in the "Vulnerabilities" column for all those software items?

cc @mostlikelee

[noahtalerman]

Fleet implementation question:

• [ ] Vulnerability processing changes: Surface CVEs for Linux kernels based on their x.y.x-* version. For example if there's a CVE for version 5.17.15, surface this CVE if a host has a Linux kernel w/ version 5.17.15 or 5.17.15-foo.bar~hotdog.

@mostlikelee how would we do this matching w/ Fleet's software inventory? Maybe we can look for all deb packages w/ the kernel value in the section column? (deb_packages table)

Also, I ran SELECT * FROM deb_packages on the host pictured in the comment above.

Looking at the query results filtered by name w/ 6.2.0 and section w/ kernel (CSV attached below), it looks like we're filtering out some of the packages from the software inventory. Is that right?

New Query - Results (04-08-24 05-36-24).csv

[stefanamaerz]

Hi @noahtalerman! 👋

Regarding this:

Vulnerability processing changes: Surface CVEs for Linux kernels based on their x.y.x-* version. For example if there's a CVE for version 5.17.15, surface this CVE if a host has a Linux kernel w/ version 5.17.15 or 5.17.15-foo.bar~hotdog.

I would suggest a feature flag here:

1. by default, match the Vulnerability precisely based on the OVAL feed
2. enable the fuzzy style matching (i.e. x.y.x-* as you describe) via a feature flag for customers interested in custom kernel vuln detection.

The reason I say this is that option 2 may have some probability of false positives. Option 1 should be deterministic. Forcing customers to enable the fuzzy match feature helps ensure they understand the pros/cons of the fuzzy style match and how additional triage may be required for their custom foo.bar~hotdog kernel 🌭

Thoughts on that?

That said this looks really good to me! 🎉

[noahtalerman]

I would suggest a feature flag here:

1. by default, match the Vulnerability precisely based on the OVAL feed
2. enable the fuzzy style matching (i.e. x.y.x-* as you describe) via a feature flag for customers interested in custom kernel vuln detection.

@stefanamaerz totally makes sense that some users might want a more exact match. Thanks for the feedback!

In the spirit of iteration (making smaller changes), I think we'll start w/ making option 2 the default. This way, we can ship the improvement quicker.

ensure they understand the pros/cons of the fuzzy style match

To set user's expectations, we'll clearly document how Fleet does the matching.

Later, we can always follow up and add a config option to allow exact matching.

[noahtalerman]

Hey @sharon-fdm and @mostlikelee this user story is ready for specs and estimation. I moved it to the settled column in drafting.

Please let me know if you have any questions/feedback on the plan. Thanks!

[sharon-fdm]

@noahtalerman we will estimate today. cc: @mostlikelee

[mostlikelee]

After review, the canonical OVAL feed does contain robust kernel matching support. I believe the gap in Fleet is that it currently only applies to items in the software table. Extending analyzation to the operating_systems table should be viable.

[sharon-fdm]

Estimation: If the customer runs a modified kernel and changes names/versions --> estimation is 8. If not then 5.

[mostlikelee]

@noahtalerman @sharon-fdm

There's a problem with mapping vulnerabilities to kernel packages (as opposed to mapping only the currently running kernel). Since it's possible to have multiple kernel packages installed, Fleet will map vulnerabilities to hosts that may not be running that kernel version.

Screenshot:

A couple alternatives:

1. insert a new software row which reports the current running kernel version with vulnerabilities mapped (kernel packages will continue to be unassociated with vulnerabilities)
2. only map vulnerabilities to kernel packages that are associated with the currently running kernel
3. map kernel vulnerabilities within the OS tab (maybe change the name from OS → System). We can list kernel versions instead of operating systems here for Linux. Host to OS is already a one-to-one relationship, so kernel version would fit naturally here.

[sharon-fdm]

@mostlikelee your comment makes sense. This is a product decision but I will add my opinion. I think to be persistent with other software, we report it if it's there even if not used. Hence I think reporting any kernel found on a host makes sense just because it COULD be run.

@noahtalerman your call.

[noahtalerman]

Fleet will map vulnerabilities to hosts that may not be running that kernel version.

@mostlikelee If I'm understanding correctly, a given host might have several kernel packages installed. But only one is active (running).

On the Vulnerabilities page in Fleet (and in GET /software/vulnerabilities endpoint), the user will see that host is included in the hosts count for any of it's kernel packages that have CVEs. Even if that host isn't running that specific kernel package.

If that's the case, I think this is acceptable for the first pass. The user can run a query to dig into which kernel version is active.

We have a separate "See which Linux kernel is active" (#18055) story to improve this in a future iteration.

[noahtalerman]

FYI @stefanamaerz ^

What do you think?

[stefanamaerz]

@noahtalerman Yep! Our vuln team expressed an interest to know of vulns in installed kernels, even if they are inactive. Ideally old/unused kernels are getting removed periodically via sudo apt autoremove when upgrades are performed.

[mostlikelee]

@noahtalerman @sharon-fdm

In analyzing the Ubuntu OVAL data, i've found all kernel CVEs must match 2 rules:

• a regex matching minimum version and variant (ie. <unix:os_release operation="pattern match">5.15.0-\d+(-gke|-ibm)</unix:os_release>)
• a minimum version (ie. <ind:value datatype="debian_evr_string" operation="less than">0:5.15.0-1003</ind:value>)

The minimum version DOES vary based on the kernel variant, which means supporting custom kernels adds some complication in order to avoid a large amount of false positives.

The current PRs for this feature implement the default OVAL matching strategies, but we need to discuss the further work to support custom kernel variants. I suggest we break that effort off into another story to unblock this story and give us time to strategize.

@stefanamaerz I'm curious if all of your kernels are branched from -generic or another variant. One possibility could be to assume unknown kernel variants map to -generic or possibly a user specified variant.

[sharon-fdm]

I support splitting this story to deliver the generic kernels vuln detection for 4.51 and discuss how we want to address custom variants.

@noahtalerman this will be your call, but we are not sure at the moment how specific kernels look with regard to the variant and version. Will wait to hear from @stefanamaerz for more info.

[noahtalerman]

kernels are branched from -generic or another variant.

UPDATE: Let's see if can get an answer from the customer (noahtalerman 2024-05-28)

~~> supporting custom kernels adds some complication in order to avoid a large amount of false positives.~~

~~@mostlikelee how noisy are we talking? False positives for custom kernels? False positives for non-custom kernels?~~

~~I think we want to lean being noisier in the first pass even if it means creating false positives.~~

~~Why? Customers would rather know about CVEs than not know. If I'm understanding correctly, if we pushed support for custom kernel variants then we wouldn't be detecting CVEs for a large number of kernels.~~

~~We can also learn/iterate faster when we hear about false positives.~~

~~@mostlikelee let me know if you have any concerns with this approach.~~

~~cc @sharon-fdm~~

[stefanamaerz]

So I did some digging: I agree with @mostlikelee's idea of splitting into 2 github issues. The more I think about some of the nuance the harder this seems to solve.

[mostlikelee]

@stefanamaerz we have an open PR #19328 which will map "unknown" kernel variants to the "-generic" variant. Would that be viable solution for you?

[noahtalerman]

We decided to make some changes to the plan:

• For this story, we'll use OVAL feed for exact matching. We won't use OVAL feed to match unknown (custom) kernels.
• In a fast follow up (separate story here), we'll use NVD for matching for unknown (custom) kernels.

cc @stefanamaerz @sharon-fdm @mostlikelee

[noahtalerman]

Heads up @Patagonia121 and @stefanamaerz, this feature was shipped in Fleet 4.51

Docs PR is merged.

[fleet-release]

Ubuntu kernel flaws, Fleet now reveals with grace, Safe in cloud's embrace.