grype
grype copied to clipboard
anchore
Distro tests fail within nixos
What happened: When building the latest version of grype the distro tests fail
What you expected to happen: Successful build of grype
How to reproduce it (as minimally and precisely as possible):
nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
Anything else we need to know?:
Test Failures:
--- FAIL: Test_NewDistroFromRelease_Coverage (0.01s)
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/amazon (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/amazon
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/busybox (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/busybox
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/debian (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/debian
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/fedora (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/fedora
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/redhat (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/redhat
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/ubuntu (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/ubuntu
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/oraclelinux (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/oraclelinux
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/custom (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/custom
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/opensuse-leap (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/opensuse-leap
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/sles (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/sles
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/photon (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/photon
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/arch (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/arch
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/partial-fields/missing-id (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/partial-fields/missing-id
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/partial-fields/unknown-id (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/partial-fields/unknown-id
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos6 (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos6
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos5 (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/centos5
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/mariner (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/mariner
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/rockylinux (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/rockylinux
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/almalinux (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/almalinux
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/gentoo (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/gentoo
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/wolfi (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/wolfi
Messages: empty linux release info
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard (0.00s)
distro_test.go:254:
Error Trace: /build/source/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
Messages: empty linux release info
distro_test.go:272: defined: ubuntu
distro_test.go:272: defined: centos
distro_test.go:272: defined: amazonlinux
distro_test.go:272: defined: oraclelinux
distro_test.go:272: defined: sles
distro_test.go:272: defined: wolfi
distro_test.go:272: defined: fedora
distro_test.go:272: defined: opensuseleap
distro_test.go:272: defined: chainguard
distro_test.go:272: defined: redhat
distro_test.go:272: defined: alpine
distro_test.go:272: defined: photon
distro_test.go:272: defined: gentoo
distro_test.go:272: defined: debian
distro_test.go:272: defined: busybox
distro_test.go:272: defined: archlinux
distro_test.go:272: defined: mariner
distro_test.go:272: defined: rockylinux
distro_test.go:272: defined: almalinux
distro_test.go:277: distro coverage incomplete (defined=19, coverage=0)
Environment:
- Output of
grype version: - OS (e.g:
cat /etc/os-releaseor similar):
ANSI_COLOR="1;34"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="24.05.20240329.d8fe5e6"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
IMAGE_ID=""
IMAGE_VERSION=""
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 24.05 (Uakari)"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="24.05 (Uakari)"
VERSION_CODENAME=uakari
VERSION_ID="24.05"
Hi @JordanFaust,
Would you mind providing more detail in your repro steps? I have never used Nix before, and I don't know how to get into a situation where I can run nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype. Can you provide a Dockerfile or set of steps that can take me from "this machine has never run nix before at all" to "I can watch this error happen?" I'd prefer a Dockerfile specifically if that's possible.
I tried the following:
- Run
docker run -ti ghcr.io/nixos/nix- this puts me at a shell in Nix - In the container run
nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
This fails with so error: experimental Nix feature 'nix-command' is disabled; add '--extra-experimental-features nix-command' to enable it. I've tried a few things, but I haven't gotten as far as the error you're seeing.
Thanks!
Unfortunately it looks like this is a specific issue for NixOS and not nix the package manager. The above runs a container with a different OS and just has the nix CLI. I'm not sure the best way to test that in a container. I'm not sure what the specific issue is but something about these tests are failing when being built within NixOS. If you have any ideas on what could be the source I can try it out myself and try building from there.
I wasn't able to reproduce this issue:
FROM ubuntu:22.04
# install prerequisites
RUN apt-get update && apt-get install -y curl sudo xz-utils
# create a non-root user (recommended for running Nix)
RUN useradd -m nixuser && echo 'nixuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
# switch to the non-root user
USER nixuser
WORKDIR /home/nixuser
# install Nix
RUN curl -L https://nixos.org/nix/install | sh
# set up Nix environment
ENV USER nixuser
ENV HOME /home/nixuser
ENV PATH /home/nixuser/.nix-profile/bin:/home/nixuser/.nix-profile/sbin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/run/current-system/sw/bin:/home/nixuser/.local/bin:$PATH
RUN . /home/nixuser/.nix-profile/etc/profile.d/nix.sh
# enable experimental features in Nix configuration
RUN mkdir -p /home/nixuser/.config/nix
RUN echo "experimental-features = nix-command flakes" >> /home/nixuser/.config/nix/nix.conf
Then:
$ docker build -t localhost/nix:latest .
$ docker run --rm -it localhost/nix:latest
And from within the container, everything seems to work:
$ docker run --rm -it localhost/nix:latest
nixuser@78cbeeb35219:~$ nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
nixuser@78cbeeb35219:~$ grype alpine:latest
✔ Vulnerability DB [updated]
✔ Parsed image sha256:ace17d5d883e9ea5a21138d0608d60aa2376c68f616c55b0b7e73fba6d8556a3
✔ Cataloged contents a0264d60f80df12bc1e6dd98bae6c43debe6667c0ba482711f0d806493467a46
├── ✔ Packages [15 packages]
├── ✔ File digests [80 files]
├── ✔ File metadata [80 locations]
└── ✔ Executables [17 executables]
✔ Scanned for vulnerabilities [14 vulnerability matches]
├── by severity: 0 critical, 0 high, 12 medium, 0 low, 0 negligible (2 unknown)
└── by status: 2 fixed, 12 not-fixed, 0 ignored
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
busybox 1.36.1-r15 apk CVE-2023-42366 Medium
busybox 1.36.1-r15 apk CVE-2023-42365 Medium
busybox 1.36.1-r15 apk CVE-2023-42364 Medium
busybox 1.36.1-r15 apk CVE-2023-42363 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42366 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42365 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42364 Medium
busybox-binsh 1.36.1-r15 apk CVE-2023-42363 Medium
libcrypto3 3.1.4-r5 3.1.4-r6 apk CVE-2024-2511 Unknown
libssl3 3.1.4-r5 3.1.4-r6 apk CVE-2024-2511 Unknown
ssl_client 1.36.1-r15 apk CVE-2023-42366 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42365 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42364 Medium
ssl_client 1.36.1-r15 apk CVE-2023-42363 Medium
A newer version of grype is available for download: 0.77.4 (installed version is 0.76.0)
Are there other specific details that I'm missing from this setup?
I don't believe there is an issue for nix the package manager. This specifically is an issue with NixOS and something specific about the nature of the distro tests that are failing within that hermetic build environment. I will need to dig into what the distro tests are actually doing to simulate the different distros. I'm assuming something in the test setup is causing issues within the nixos build environment. I can try and track this down if you can help me understand these tests a little better.
Thanks @JordanFaust !
What the "distro" tests are doing is testing Grype's ability to detect what distro the image it's scanning is based on.
The tests are defined here: https://github.com/anchore/grype/blob/834793100e7dc1455b2b7b9998a10434d99bf6e9/grype/distro/distro_test.go#L109
My guess is that something in here is broken by the hermetic build environment.
The way the tests essentially work is:
- There's a test fixture that looks the identifying files on the distro, for example https://github.com/anchore/grype/blob/main/grype/distro/test-fixtures/os/alpine/etc/os-release looks like
/etc/os-releasefrom an Alpine image. - The test makes a source and file resolver from this directory - the source and file resolver are grype's abstraction over file I/O.
- The tests exercise Grype's distro identification code by passing it the resolver.
They're failing at require.NotNil(t, release, "empty linux release info"), which means that given the file resolver, Grype couldn't identify the distro. I suspect this is because something about the hermetic build prevents Grype from doing the necessary file I/O or something like that.
Creating the file resolver is happening over in Syft code here: https://github.com/anchore/syft/blob/2d318cffaaca07a8b3e87d68f80f881153a305b8/syft/source/directorysource/directory_source.go#L135
which calls: https://github.com/anchore/syft/blob/2d318cffaaca07a8b3e87d68f80f881153a305b8/syft/internal/fileresolver/directory.go#L40
I don't see anything in there that I think NixOS should be blocking - the chroot-named functions don't seem to be actually making a chroot system call, just returning a file resolver rooted outside the current directory.
Thanks for taking a look! Let me know if there's any other help I can be.
Hello all, I help maintain the syft and grype packages in nixpkgs. I'm looking into this as part of our Zero Hydra (our ci system) Failures process for the next stable release.
https://hydra.nixos.org/build/259554448/nixlog/3 This is the tests running within the build sandbox.
I've tried just running go as normal on my NixOS machine and also get the issue
I have trimmed the logs as they're nothing new.
$ go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...
=== RUN Test_NewDistroFromRelease_Coverage
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
distro_test.go:254:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
Messages: empty linux release info
#...
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
distro_test.go:254:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:254
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
Messages: empty linux release info
=== NAME Test_NewDistroFromRelease_Coverage
distro_test.go:272: defined: archlinux
#...
distro_test.go:272: defined: debian
distro_test.go:277: distro coverage incomplete (defined=19, coverage=0)
--- FAIL: Test_NewDistroFromRelease_Coverage (0.01s)
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine (0.00s)
#...
--- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard (0.00s)
FAIL
FAIL github.com/anchore/grype/grype/distro 0.013s
FAIL
Interestingly in a container I don't have the issue
docker run -it nixos/nix nix develop --experimental-features "nix-command flakes" nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
Will put you in a shell with the dependencies used at build time
You can then run phases="unpackPhase patchPhase" genericBuild to get you into a situation where the source code is available in $PWD
Then you can run go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...
And I'm not getting the issue inside the nixos/nix docker container.
Is there an easy way to make the logs visible so I can check the errors during the linux.IdentifyRelease() call? Just seeing nil doesn't narrow it down much.
If the logs aren't easily visible I'll probably make it return the error
Thanks for taking a look @06kellyjac!
I haven't found an easy way to enable Grype's normal logging during a test run, so it might make sense to do a quick local change to return the error.
Here's what's needed to log within tests (today... we can probably make this better):
import (
"github.com/anchore/go-logger"
"github.com/anchore/go-logger/adapter/discard"
"github.com/anchore/go-logger/adapter/logrus"
"github.com/anchore/grype/internal/log"
"github.com/anchore/syft/syft"
...
)
...
for _, test := range tests {
t.Run(test.fixture, func(t *testing.T) {
l, err := logrus.New(
logrus.Config{
EnableConsole: true,
Level: logger.TraceLevel,
},
)
syft.SetLogger(l)
log.Set(l)
require.NoError(t, err)
require.NotNil(t, l)
// run your test...
syft.SetLogger(discard.New())
log.Set(discard.New())
})
}
Another observation, the link to the logs doesn't seem to be running tests with -v, which means that stdout will be suppressed. So you'll need to make both changes to see the log output in your CI environment.
Running locally with those changes:
I don't see anything super useful here from my understanding. Unless maybe it sees "ignoring /" and that ignores everything? But I wouldn't expect that as the test fixture dir is meant to be the root from that point right?
I don't see any of the error traces so unless the logging isn't set up right I guess it's not running into problems there
=== RUN Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
[0000] WARN no explicit name and version provided for directory source, deriving artifact ID from the given path (which is not ideal)
[0000] DEBUG ignoring system mountpoint mountpoint=/
[0000] DEBUG ignoring system mountpoint mountpoint=/run
[0000] DEBUG ignoring system mountpoint mountpoint=/dev
[0000] DEBUG ignoring system mountpoint mountpoint=/dev/shm
[0000] DEBUG ignoring system mountpoint mountpoint=/proc
[0000] DEBUG ignoring system mountpoint mountpoint=/run/wrappers
[0000] DEBUG ignoring system mountpoint mountpoint=/sys
[0000] DEBUG ignoring system mountpoint mountpoint=/run/user/1000
[0000] TRACE indexing filetree path=test-fixtures/os/chainguard
distro_test.go:268:
Error Trace: /home/jk/projects/personal/grype/grype/distro/distro_test.go:268
Error: Expected value not to be nil.
Test: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
Messages: empty linux release info
=== NAME Test_NewDistroFromRelease_Coverage
Just running locally in my case I've got a tmpfs on root. Not 100% sure for the CI. There's a non-insignificant amount of NixOS users following this pattern to identify where their state is.
https://grahamc.com/blog/erase-your-darlings/ https://github.com/nix-community/impermanence
[0000] DEBUG ignoring system mountpoint mountpoint=/
ha! Agreed, that one seems a tad impactful (that will ignore the entire filesystem I believe). Syft doesn't catalog within certain filesystem mounts, so it seems like there is something peculiar with the system mounts (or something that needs to be corrected in syft).
Can you confirm on a system that is exhibiting the bad behavior what the filesystem types for each mount? Would you be willing to post the output of df from that system? (or a portion of it?)
edit: I do see a hint from the links you posted that we know the answer already...
If you have a lot of RAM, you could skip the erase step and make / a tmpfs.
I think this issue has the same root cause as https://github.com/anchore/syft/issues/2894, which is more obviously a bug in Syft.
Hi @JordanFaust and @06kellyjac I believe https://github.com/anchore/syft/pull/2918, released in https://github.com/anchore/syft/releases/tag/v1.6.0 (which is now a couple behind - latest is https://github.com/anchore/syft/releases/tag/v1.8.0) fixed this. Would you mind re-testing? Please let us know if you're still facing this issue.
I can confirm things are working. I can successfully build grype within nixos now