unable to initialize a remote scanner for some images

[seekermarcel]

What steps did you take and what happened:

I have a running trivy-operator in an air-gapped environment. All registries are configured to use a private harbor instead of docker.io or github. I use the internal server mode and there is an istio mesh in between all communications except the trivy-operator and the trivy-server. (The scan jobs do not have istio at all)

For a very small amount of workloades i get the following error:

{"level":"error","ts":"2024-02-23T11:17:57Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-b6b6c4c97","container":"istio-proxy","status.reason":"Error","status.message":"2024-02-23T11:17:54.346Z	FATAL	image scan error: scan error: unable to initialize a scanner: unable to initialize a remote image scanner: 4 errors occurred:
	* docker error: unable to inspect the image (auto): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
	* containerd error: containerd socket not found: /run/containerd/containerd.sock
	* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
	* remote error: GET https://index.docker.io/v2/library/auto/manifests/latest: UNAUTHORIZED: authentication required;

Most images are no problem at all and get scanned and reported normally.

What did you expect to happen:

I expected a valid scan and a report at the end like for most other images.

Anything else you would like to add:

What are remote image scanners and why is it trying to initialize something like that? There shouldn't be anything remote especially in an air-gapped env

Environment:

• Trivy-Operator version (use trivy-operator version): v0.20.6
• Kubernetes version (use kubectl version): Server 1.27.6

[seekermarcel]

The problem seems to be a mechanic of istiod. As a sidecar istio searches for the keyword "auto" and replaces it with the actual image. The image "auto" however does not exist for trivy and isn't replaced in the statefullset before scanning. Thereby it can't scan istio sidecars comming from replicasets

[chen-keinan]

@seekermarcel can you please share an example of descriptor with the "auto" keyword. in addition do you have any suggestion on how to solve it ?

[seekermarcel]

sure. I hope the following is enough to understand. The problem exists for all resources that want to do image replacement. As an idea i would propose not to scan deployments, statefulsets, replicasets etc that contain image: auto but instead only their pods 🤔

containers:
- image: auto
  imagePullPolicy: IfNotPresent
  name: istio-proxy
  ports:
  - containerPort: 15090
    name: http-envoy-prom
    protocol: TCP
  resources:
    limits:
      cpu: 800m
      memory: 384Mi
    requests:
      cpu: 140m
      memory: 320Mi
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsGroup: 1337
    runAsNonRoot: true
    runAsUser: 1337
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File

[chen-keinan]

@seekermarcel the down side of it is that if a replicaSet is set to 4 then instead of one report for the container you'll get 4 exact reports for the same container.

Related #1745

[alfsch]

The compare solution from #1745 seems to be the solution, which would work in all use cases of service injection and it solves also troubles when image mutations (kyverno) are used.

[chen-keinan]

@alfsch could you elaborate more on your suggestion above ?

[alfsch]

@chen-keinan when you have a service mesh like istio or other services which change the image name with mutation hooks you run into troubles when you are scanning the deployment or the replicaset and other high level resources.

A snippet in istio case from deployment or replicaset defines the main container image like that:

containers:
- image: auto

This container image is replaced by a mutating hook with the real image in the pod definition

containers:
- image: istio/proxyv2:1.20.2-distroless

The problem is that trivy run's into scan errors, when it tries to get the image with the name auto (extracted from deployment or replicaset) from docker.io. An image with the name auto doesn't exist there.

A solution could be to look into the pod's generated from the high level resources for the images (init-containers, containers, sidecars (new in k8s 1.29)) and use the found images for scanning and store this into the result for the deployment, statefulset,deamonset.

The truth is always only in the pods. If you scan only the things you find in the high level resources, you can get green scans even if harmful images are injected into the pods.

In the istio case, if you scan the deployment only, you miss 3 images to scan (2 istio init container and one sidecar container)

[github-actions[bot]]

This issue is stale because it has been labeled with inactivity.