Trivy Server Integration Reports Different Vulnerabilities Compared to Trivy Client

[cheapshot2000]

Current Behavior

Dependency Track, using ONLY Trivy Server integration, reports different vulnerabilities that the Trivy Client/CLI does with the same SBOM.

Ultimately, we are trying to get scans from our CICD pipelines and DT to align.  However, DT is processing / scanning differently using the Trivy Server integration resulting in vastly different vulnerability results than using the Trivy client. Even with no other analyzers, scanners, or data sources beyond the trivy server integration.

Steps to Reproduce

1. Setup DT in an ‘offline’ style mode – this is to ensure only Trivy information is displayed in DT.

• No NIST data,
• No internal analyzers,
• No ability to reach the internet.

2. Configure Trivy Server Analyzer Integration - we are using Trivy 0.55
3. Obtain the redhat/ubi9:latest container image.
4. Using the Trivy CLIENT, generate an SBOM of the UBI image.
5. Also, using Trivy CLIENT, scan the generated SBOM for vulnerabilities.

• Note the quantity of vulnerabilities (191 in this example)

6. Upload the SBOM to offline DT w/ Trivy Server integration.

• Note the quantity of vulnerabilities reported from Trivy Server (265 in this example)

Files for reproducing:::

Export of the UBI from DT w/ Trivy Server Integration ubi9_bom_exported_from_DT.json

Trivy Client output showing far fewer vulnerabilities ubi9_vuln_scan.txt

Trivy Client SBOM w/ Vulns ubi9_sbom_with_vulns.cdx.json

Trivy Client SBOM w/o Vulns ubi9_sbom_without_vulns.cdx.json

Expected Behavior

When DT is ‘offline’ and with no other data sources, the vulnerabilities reported by DT w/ Trivy Server integration should match that from Trivy Client.

I understand that DT has other datasources and will decorate/assess differently. The offline aspect of this is to show that this is only coming from the trivy server integration with DT - and that differences are not coming from other data sources.

Dependency-Track Version

4.11.7

Dependency-Track Distribution

Container Image

Database Server

PostgreSQL

Database Server Version

14.12

Browser

Google Chrome

Checklist

• [X] I have read and understand the contributing guidelines
• [X] I have checked the existing issues for whether this defect was already reported

[cheapshot2000]

Just seeing if anyone has any thoughts on what is causing this or possible workarounds?

[nscuro]

It's likely that we're currently not (or wrongly) populating a Trivy server API field that helps Trivy in "narrowing down" on the actually applicable vulnerabilities.

Seeing that you're dealing with UBI images, my gut feeling is that we're not properly communicating that they are Red Hat components, so Trivy cannot take Red Hat advisories into consideration. Again, just a gut feeling, would need to verify.

[Shadow-Templar]

Is there any update to this, or has it been verified what the likely culprit could be?

[AkselAllas]

I am experiencing the same thing. My images are debian bookworm.

[AkselAllas]

I wasted some time here with Wireshark. (I was hoping to connect proto files and see protobuf unmarshalled, but Wireshark only supports that for UDP or HTTP/2 sadly. Would make much more sense to print unmarshaled protobuf in dev application setup)

One difference I found between sending locally using trivy cli to trivy cli server and sending to dependency track trivy integration via dependency track frontend is as follows:

In protobuf call from dependency track to Trivy integration Scanners field of ScanOptions of POST /twirp/trivy.scanner.v1.Scanner/Scan HTTP/1.1 is set as os, library, vuln

In protobuf call from trivy cli to trivy cli server Scanners field of ScanOptions of POST /twirp/trivy.scanner.v1.Scanner/Scan HTTP/1.1 is set as os, library, vuln, sbom

[AkselAllas]

Another potential difference 1 vs 2 scans 🤔

Trivy CLI:

Dependency Track Integration

I initially thought that OS is being set wrongly, but both had Debian 12.7 set.

Dependency Track case with 2 scans had:

1. PutBlob python-pkg and
2. PutBlob Debian 12.7

[AkselAllas]

@nscuro Perhaps this should be a P1 issue. This kind of makes the Trivy integration unusable.

I have hundreds of noisy/non-applicable Critical + High issues. Too many to manually suppress.

[nscuro]

I will try to have a look this weekend. There are a few Protobuf extensions for BurpSuite, I am optimistic that will yield some insights.

[nscuro]

I think I found the issue(s), they should be fixed with https://github.com/DependencyTrack/dependency-track/pull/4245.

@cheapshot2000 with those fixes, I'm now getting 197 vulnerabilities for the BOM you provided. I haven't done a diff of the vulns, but I suspect the additional 6 are due to them having been added since you originally reported the issue.

Once the linked PR is merged to master, you will be able to test it using the dependencytrack/apiserver:snapshot container image. You can use the dev Compose files for that purpose:

cd dev
docker compose -f docker-compose.yml -f docker-compose.trivy.yml up -d --pull always

[cheapshot2000]

Great news! Thank you @nscuro . We'll give it a try early next week...

[AkselAllas]

@nscuro Thanks for the improvement!

But still there are differences in Trivy Client and Dependency Track integration.

I tried sbom scan of ubi9_sbom_with_vulns.cdx.json using the correct snapshot image (dependencytrack/apiserver@sha256:cf46b02fb8368823f68fa91e41ea7ad7746c738b258888035d7cb0ecd305012c)

With Trivy client there are 197 vulnerabilities (0 criticals, 2 high) With Dependenct Track there are 197 vulnerabilities (5 criticals, 88 high)

I will also send you a bom on Slack that shows 3 criticals, 54 high in Trivy client, but 9 criticals, 83 high in Dependency Track

[nscuro]

Severity and CVSS vectors are taken as-is from the Trivy response: https://github.com/DependencyTrack/dependency-track/blob/5235dd869afb3150da557642f821bb85de4e41fe/src/main/java/org/dependencytrack/parser/trivy/TrivyParser.java#L71-L94

Do note though that if you have mirrored CVEs from the NVD (which is enabled by default), DT will not let Trivy overwrite CVE data, which includes severity. So it's very possible the difference you're seeing is caused by DT showing severities sourced from the NVD, whereas Trivy might've used a vendor-specific rating.

[AkselAllas]

In this case, Trivy Severity and CVSS are much more useful.

I tested by disabling NVD mirroring in the snapshot release, re-imported the BOM and got the correct (3 critical, 54 high) for my bom.

What would the downsides of disabling NVD mirroring be exactly?

• If I disabled mirroring, would I still have the correct vulnerability entities in the DB. e.g: Could I still triage and suppress vulns per project?

[AkselAllas]

Also, any tips on how to disable snapshot popup on every refresh of the frontend?

Frontend code is:

    if (this.$dtrack && this.$dtrack.version.includes('SNAPSHOT')) {
      this.$root.$emit('bv::show::modal', 'snapshotModal');
    }

So I could change unzip the jar from the docker, change version and re-jar it.

But I think it would be best to do a minor release of dependency-track with this fix attached.

[AkselAllas]

Can we copy this functionality from Snyk to Trivy?

[nscuro]

@AkselAllas But I think it would be best to do a minor release of dependency-track with this fix attached.

If you look at the PR I referenced, it has a backport/4.12.1 label so it will be backported to the next bugfix release.

Can we copy this functionality from Snyk to Trivy?

No. The reason it works for Snyk is because Snyk has its own vulnerabilities (SNYK-xxxx). Since Trivy uses vulnerability identifiers of other public sources (CVE, GHSA), we cannot apply the same logic for it. We're planning to improve this situation (and possibly make the Trivy integration obsolete) via #4122, but it's a larger chunk of work.

[cheapshot2000]

@nscuro - Thank you for the fix; just wanted to confirm with the snapshot version mentioned above we are seeing identical vuln #'s from trivy client compared to trivy/dt integration. :)

[nscuro]

Thanks @cheapshot2000! Closing this issue then. We're hoping to get the v4.12.1 release out this week.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.