trivy icon indicating copy to clipboard operation
trivy copied to clipboard

Vulnerabilities show and disappear intermittently and without reason

Open tmetodie opened this issue 3 years ago • 14 comments

Description

Started using Trivy for our images while migrating from microscanner. However when I run the tests on the repository, I see the following result.

73523cf38c16 (alpine 3.9.4)
===========================
Total: 6 (UNKNOWN: 2, LOW: 0, MEDIUM: 4, HIGH: 0)

+------------+------------------+----------+-------------------+---------------+--------------------------------+
|  LIBRARY   | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+------------+------------------+----------+-------------------+---------------+--------------------------------+
| curl       | CVE-2020-8169    | MEDIUM   | 7.64.0-r3         | 7.64.0-r4     | libcurl: partial *** leak |
|            |                  |          |                   |               | over DNS on HTTP redirect      |
+            +------------------+          +                   +               +--------------------------------+
|            | CVE-2020-8177    |          |                   |               | curl: Incorrect argument check |
|            |                  |          |                   |               | can allow remote servers to    |
|            |                  |          |                   |               | overwrite local files...       |
+------------+------------------+          +                   +               +--------------------------------+
| libcurl    | CVE-2020-8169    |          |                   |               | libcurl: partial *** leak |
|            |                  |          |                   |               | over DNS on HTTP redirect      |
+            +------------------+          +                   +               +--------------------------------+
|            | CVE-2020-8177    |          |                   |               | curl: Incorrect argument check |
|            |                  |          |                   |               | can allow remote servers to    |
|            |                  |          |                   |               | overwrite local files...       |
+------------+------------------+----------+-------------------+---------------+--------------------------------+
| musl       | CVE-2020-28928   | UNKNOWN  | 1.1.20-r5         | 1.1.20-r6     |                                |
+------------+                  +          +                   +               +--------------------------------+
| musl-utils |                  |          |                   |               |                                |
+------------+------------------+----------+-------------------+---------------+--------------------------------+

After a rerun but moving an ENV and ARG lines inside the Dockerfile, no vulnerabilities are show:

88aa7757665e (alpine 3.9.4)
===========================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0)

Same behavior for filesystem and image scan. trivy filesystem --exit-code 0 --severity UNKNOWN,LOW,MEDIUM,HIGH --vuln-type os,library --no-progress / trivy filesystem --exit-code 1 --severity CRITICAL --no-progress /

trivy image --exit-code 0 --severity UNKNOWN,LOW,MEDIUM,HIGH --vuln-type os,library --no-progress trivy image --exit-code 1 --severity CRITICAL --no-progress

Changes to Dockerfile

FROM node:11-alpine

- RUN apk update && apk upgrade && apk add curl
ARG version
ENV VERSION_NUMBER=$version

+ RUN apk update && apk upgrade && apk add curl && \
    mkdir -p /app/src
- RUN mkdir -p /app/src

What did you expect to happen?

Same vulnerabilities should be shown as no chance in regards to the vulnerabilities was performed.

What happened instead?

At some point vulnerabilities are not displayed anymore.

Output of run with -debug:

(paste your output here)

Output of trivy -v:

Version: 0.13.0
Vulnerability DB:
  Type: Light
  Version: 1
  UpdatedAt: 2020-11-23 12:16:02.670235969 +0000 UTC
  NextUpdate: 2020-11-24 00:16:02.670235469 +0000 UTC
  DownloadedAt: 2020-11-23 13:22:16.881275101 +0000 UTC

Additional details (base image name, container registry info...):

tmetodie avatar Nov 23 '20 13:11 tmetodie

@simar7 Could you have a look at it?

knqyf263 avatar Dec 15 '20 09:12 knqyf263

Hello Team! Were you able to find anything related to this one?

tmetodie avatar Jan 21 '21 19:01 tmetodie

I'm afraid but I didn't have time before or now to look at it.

simar7 avatar Jan 21 '21 20:01 simar7

This issue is stale because it has been labeled with inactivity.

github-actions[bot] avatar Mar 23 '21 00:03 github-actions[bot]

@tmetodie are you still with problems?

krol3 avatar Mar 23 '21 03:03 krol3

We see this quite often. We scan all our Docker images, and vulnerabilities in the base image suddenly resolve without the base image being changed. Or, likewise, one image reports a vulnerability but the other one doesn't, although the base image is again the same.

fxnn avatar Apr 23 '21 06:04 fxnn

Hello guys, do you have any information about this issue? I integrated trivy v 0.23.0 in my CI/CD and i have the same issue. In a run, trivy reports CVE's, after a re-run reports no CVE's anymore.

alecsz avatar Feb 11 '22 11:02 alecsz

First of all, rebuilding a container image may update package versions if you don't pin those versions. But it would be weird if you scan the same image with the same digest. @afdesk Can you take a look?

knqyf263 avatar Feb 13 '22 08:02 knqyf263

@knqyf263 you are right, but what i mean is using trivy fs scan to scan the file system, in my case the pom file, before i build the image.

Pom file wasn't changed.

I reproduced this bug by re-run a few times and the result is sometimes with vulnerabilities sometimes without any vulnerabilities. As i said the file system(in this case relevant file is the pom file) wasn't changed.

alecsz avatar Feb 14 '22 08:02 alecsz

@alecsz thanks for your report! I'll try to clarify this issue.

afdesk avatar Feb 14 '22 13:02 afdesk

@alecsz I wanted to reproduce a problem.

  1. I've created demo pom.xml with log4j module: https://gist.github.com/afdesk/b51b393bd99fd12a7a7fa30efc093e4a
  2. Run trivy in debug mode:
$ trivy -d fs .
...
2022-02-14T19:25:42.983+0600	DEBUG	Vulnerability type:  [os library]
2022-02-14T19:25:42.984+0600	DEBUG	Resolving org.apache.logging.log4j:log4j:2.0.2...
2022-02-14T19:25:43.551+0600	DEBUG	Analysis error: failed to parse pom.xml: module error (log4j-api): unable to open the relative path: stat log4j-api: no such file or directory
2022-02-14T19:25:43.554+0600	DEBUG	OS is not detected and vulnerabilities in OS packages are not detected.
2022-02-14T19:25:43.554+0600	DEBUG	Detected OS: unknown
2022-02-14T19:25:43.555+0600	INFO	Number of language-specific files: 0
...
  1. but when I run trivy in offline scanning, I can see vulenrabilites:
$ trivy -d fs --offline-scan .
...
2022-02-14T19:33:03.644+0600	DEBUG	Resolving org.apache.logging.log4j:log4j:2.0.2...
2022-02-14T19:33:03.645+0600	DEBUG	Fetching the remote pom.xml is skipped
2022-02-14T19:33:03.645+0600	DEBUG	org.apache.logging.log4j:log4j:2.0.2 was not found in local/remote repositories
2022-02-14T19:33:03.646+0600	DEBUG	OS is not detected and vulnerabilities in OS packages are not detected.
2022-02-14T19:33:03.646+0600	DEBUG	Detected OS: unknown
2022-02-14T19:33:03.646+0600	INFO	Number of language-specific files: 1
2022-02-14T19:33:03.646+0600	INFO	Detecting pom vulnerabilities...
2022-02-14T19:33:03.646+0600	DEBUG	Detecting library vulnerabilities, type: pom, path: pom.xml

pom.xml (pom)
=============
Total: 7 (UNKNOWN: 1, LOW: 1, MEDIUM: 2, HIGH: 0, CRITICAL: 3)
...

@alecsz does it look like your issue?

afdesk avatar Feb 14 '22 13:02 afdesk

Hi @afdesk,

no this is not the issue. i added here two report files from the same pom.xml, where trivy -d fs . in one rund detects no cve and in another one cve.

report_with_cve.txt report_without_cve.txt

alecsz avatar Feb 15 '22 09:02 alecsz

@alecsz this is a strange situation. I took a look at the logs and can see two different versions of io.netty:netty-codec-http. the first version has a vulnerability, but the second was fixed.

With CVE:

DEBUG   Resolving io.netty:netty-codec-http:4.1.69.Final...

Without CVE:

DEBUG   Resolving io.netty:netty-codec-http:4.1.72.Final...

afdesk avatar Feb 18 '22 13:02 afdesk

I also see something similar when running locally on my laptop. I run trivy and it scans the file, but reports no failures. I immediately rerun the same command, and it successfully finds the failures. Very intermittent on if it comes back successful or not.


$ trivy config --severity HIGH,CRITICAL --config-policy ./trivy-policies -d ./staging/kubeseal
2022-08-04T14:38:17.767-0700	DEBUG	Severities: ["HIGH" "CRITICAL"]
2022-08-04T14:38:17.803-0700	DEBUG	cache dir:  /Users/david/Library/Caches/trivy
2022-08-04T14:38:17.804-0700	INFO	Misconfiguration scanning is enabled
2022-08-04T14:38:18.126-0700	DEBUG	OS is not detected.
2022-08-04T14:38:18.126-0700	INFO	Detected config files: 1
2022-08-04T14:38:18.126-0700	DEBUG	Scanned config file: controller.yaml

$ trivy config --severity HIGH,CRITICAL --config-policy ./trivy-policies -d ./staging/kubeseal
2022-08-04T14:38:23.193-0700	DEBUG	Severities: ["HIGH" "CRITICAL"]
2022-08-04T14:38:23.229-0700	DEBUG	cache dir:  /Users/david/Library/Caches/trivy
2022-08-04T14:38:23.229-0700	INFO	Misconfiguration scanning is enabled
2022-08-04T14:38:23.559-0700	DEBUG	OS is not detected.
2022-08-04T14:38:23.559-0700	INFO	Detected config files: 1
2022-08-04T14:38:23.559-0700	DEBUG	Scanned config file: controller.yaml

controller.yaml (rbac)

Tests: 17 (SUCCESSES: 13, FAILURES: 2, EXCEPTIONS: 2)
Failures: 2 (HIGH: 0, CRITICAL: 2)
...

Version:

$ trivy version
Version: 0.30.4

wavemoran avatar Aug 04 '22 21:08 wavemoran

I also see something similar when running locally on my laptop. I run trivy and it scans the file, but reports no failures. I immediately rerun the same command, and it successfully finds the failures. Very intermittent on if it comes back successful or not.

$ trivy config --severity HIGH,CRITICAL --config-policy ./trivy-policies -d ./staging/kubeseal
2022-08-04T14:38:17.767-0700	DEBUG	Severities: ["HIGH" "CRITICAL"]
2022-08-04T14:38:17.803-0700	DEBUG	cache dir:  /Users/david/Library/Caches/trivy
2022-08-04T14:38:17.804-0700	INFO	Misconfiguration scanning is enabled
2022-08-04T14:38:18.126-0700	DEBUG	OS is not detected.
2022-08-04T14:38:18.126-0700	INFO	Detected config files: 1
2022-08-04T14:38:18.126-0700	DEBUG	Scanned config file: controller.yaml

$ trivy config --severity HIGH,CRITICAL --config-policy ./trivy-policies -d ./staging/kubeseal
2022-08-04T14:38:23.193-0700	DEBUG	Severities: ["HIGH" "CRITICAL"]
2022-08-04T14:38:23.229-0700	DEBUG	cache dir:  /Users/david/Library/Caches/trivy
2022-08-04T14:38:23.229-0700	INFO	Misconfiguration scanning is enabled
2022-08-04T14:38:23.559-0700	DEBUG	OS is not detected.
2022-08-04T14:38:23.559-0700	INFO	Detected config files: 1
2022-08-04T14:38:23.559-0700	DEBUG	Scanned config file: controller.yaml

controller.yaml (rbac)

Tests: 17 (SUCCESSES: 13, FAILURES: 2, EXCEPTIONS: 2)
Failures: 2 (HIGH: 0, CRITICAL: 2)
...

Version:

$ trivy version
Version: 0.30.4

thanks a lot for your report! Could you test your files with the latest Trivy?

afdesk avatar Aug 23 '22 12:08 afdesk

Using version 0.31.2 appears to be returning the expected results consistently. Thanks!

wavemoran avatar Aug 23 '22 15:08 wavemoran

Using version 0.31.2 appears to be returning the expected results consistently. Thanks!

glad to read it, Thank you!

afdesk avatar Aug 24 '22 08:08 afdesk

@wavemoran please, feel free to report about any issue here!

afdesk avatar Aug 24 '22 08:08 afdesk