Not able to pull images from GCR

[evr3n]

What steps did you take and what happened:

I am trying to scan images from a private GCR registry. I created a service account and secret and added the imagePullSecret to the CM.

No vulnerability reports are generated and the trivy-operator pod log returns the following:

{"level":"error","ts":1658497288.2148783,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-7df97c8c47","container”:"XXX","status.reason":"Error","status.message":"2022-07-22T13:37:40.728Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: unable to initialize a docker scanner: 4 errors occurred:\n\t* unable to inspect the image (eu.gcr.io/PROJECT-NAME/IMAGE-NAME): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* containerd socket not found: /run/containerd/containerd.sock\n\t* GET https://eu.gcr.io/PROJECT-NAME/IMAGE-NAME: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication\n\n\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:375\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}

What did you expect to happen:

Trivy-operator scanning the images and generating vulnerability reports.

Anything else you would like to add:

We have had the same error in the past when running Trivy CLI in a pipeline and now pull the image right before the Trivy command, so it is cached. Related issues #69, https://github.com/aquasecurity/trivy-operator/issues/264 and https://github.com/aquasecurity/starboard/issues/1189

Environment:

• ArgoCD/Helm
• Trivy-Operator version (use trivy-operator version): 0.1.5
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12-gke.1500", GitCommit:"6c11aec6ce32cf0d66a2631eed2eb49dd65c89f8", GitTreeState:"clean", BuildDate:"2022-05-11T09:25:37Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): MacOS 12.4

[chen-keinan]

@evr3n thank you for opening this issue , I will review it.

[chen-keinan]

@evr3n I was able to test trivy-operator with gcr (private) and it was working ok no issues found. what I did to test it :

1. I have confirmed that json_key is good:

• cat gcp-trivy-operator-key.json | docker login -u _json_key --password-stdin https://gcr.io/

2. created secret key

• kubectl create secret docker-registry trivy-operator-gcp-access --namespace default --docker-server=gcr.io --docker-username=_json_key --docker-password="$(cat gcp-trivy-operator-key.json)" --docker-email="test-account@.iam.gserviceaccount.com"

3. updated my workload with imagePullSecret:

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: trivy-operator-gcp-access

Note: I have done another test without creating secret and without adding the imagePullSecrets to my workload and I got the same error you got above

[evr3n]

@chen-keinan thanks for your response. I did create the secret the same way, however I want to give the operator's service account the imagePullSecret, not the workload that will be scanned. Is there a way to achieve that?

[chen-keinan]

@chen-keinan thanks for your response. I did create the secret the same way, however I want to give the operator's service account the imagePullSecret, not the workload that will be scanned. Is there a way to achieve that?

In a similar way:

• create service account with imagePullSecret

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2015-08-07T22:02:39Z
  name: gcp-sa
  namespace: default
  uid: 052fb0f4-3d50-11e5-b066-42010af0d7b6
imagePullSecrets:
- name: trivy-operator-gcp-access

• add service account name to podSpec

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  serviceAccountName: gcp-sa

[evr3n]

I think we might be misunderstanding each other, so just to make sure let me summarize: we have a service account for the trivy-operator which holds the imagePullSecret, as we have a very large number of workloads to scan and don't want to change their spec if avoidable. So we want to specify the imagePullSecret on the operator's side and not on the workloads' side.

[chen-keinan]

Thank you for the clarification.

The way trivy-operator works today , it find references to image pull secrets (direct references from pod spec or indirect via service account )

• direct: pod spec --> image pull secret -- > secret
• indirect: pod spec --> service account name -->service account--> image pull secret --> secret

see reference to private registries

[dkarnutsch]

@evr3n I stumbled across this thread, because I had the same problem...

Your pullsecret might be of type kubernetes.io/dockercfg, which is considered legacy. It should use kubernetes.io/dockerconfigjson instead. Relevant lines of code: https://github.com/aquasecurity/trivy-operator/blob/6276bb14942ec0d45b78c44cdf20b4fb43f5d86d/pkg/kube/secrets.go#L63

[JAORMX]

I'm actually struggling with a similar issue. Except that the secrets in my cluster are kubernetes.io/dockerconfigjson 😕

[chen-keinan]

Fixed , now supported with global secrets