Vulnerability scan error

[huydq2vietcapitalbank]

Describe the bug When scanning K8S server, the vulnerability scan shows error.

To Reproduce Steps to reproduce the behavior:

1. Go to 'Topology'
2. Click on 'Table'
3. Tick K8S server -> Actions -> Start vulnerability scan
4. See error after a while

Screenshots

Components/Services affected

• [ ] Agent

Agent log file time="2022-10-03 08:29:14" level=error msg="error from syft command for syftArgs: packages dir:/fenced/mnt/host/ -o json --file /tmp/RFSegi6bqrYSoutput.json -q --exclude ./var/lib/docker/** --exclude ./var/lib/containerd/** --exclude ./mnt/** --exclude ./run/** --exclude ./proc/** --exclude ./dev/** --exclude ./boot/** --exclude ./home/kubernetes/containerized_mounter/** --exclude ./sys/** --exclude ./lost+found/** --exclude ./dev/** --exclude ./sys/fs/cgroup/** --exclude ./dev/shm/** --exclude ./run/containerd/containerd.sock/** --exclude .//dev/shm/** --exclude .//sys/fs/cgroup/** --exclude .//run/** --exclude .//run/user/1004/** --exclude .//var/lib/docker/containers/3eb9275b5f22e96878732f10a7660bf2ff59c2e1392fcc1dcff6e847fb956aed/mounts/shm/** --exclude .//var/lib/docker/containers/2bcac206e326756db56a7e19534c086f3a77a677d49f0342d0179f0a3b7a7e95/mounts/shm/** --exclude .//var/lib/docker/containers/32ea021ff67c756c4737893c120d3e8938b1b626247a0e244a0f7f58b70cf43b/mounts/shm/** --exclude .//var/lib/kubelet/pods/e044769e-02b2-4946-baa2-edec9600009c/volumes/kubernetes.io~secret/cattle-credentials/** --exclude .//var/lib/kubelet/pods/23b0dd90-9eea-4f7d-adc2-585101321ea5/volumes/kubernetes.io~secret/kube-proxy-token-q868q/** --exclude .//var/lib/kubelet/pods/ab313be5-d1ad-4ce4-8e50-7af7e9e16120/volumes/kubernetes.io~secret/nfs-server-nfs-server-provisioner-token-2xs6q/** --exclude .//var/lib/kubelet/pods/e044769e-02b2-4946-baa2-edec9600009c/volumes/kubernetes.io~secret/cattle-token-jwxxw/** --exclude .//var/lib/kubelet/pods/ce10f4fd-e004-4011-bc49-d17d4313e8c0/volumes/kubernetes.io~secret/exporter-node-cluster-monitoring-token-zmptv/** --exclude .//var/lib/kubelet/pods/b6649757-0112-4ff0-8ed5-44ddb7c699fa/volumes/kubernetes.io~secret/default-token-zhjdd/** --exclude .//var/lib/kubelet/pods/ce692a9c-baef-4fcc-9472-878d1c595141/volumes/kubernetes.io~secret/calico-node-token-2l52n/** --exclude .//var/lib/docker/containers/73b3a95eade2cbd7c85f46ecb81d52de85a5722706f42c25f4dfc22ffc3bf6fd/mounts/shm/** --exclude .//var/lib/docker/containers/a0f576852f44938858a2d2c1e14fb8059073135a7540ecb9bcd225b554412d3d/mounts/shm/** --exclude .//var/lib/docker/containers/44d256df7960ed4e4f84dcb042b29f30ff22fb62d868b66cf4e1d49f40c020b2/mounts/shm/** --exclude .//var/lib/docker/containers/6d0416aa4cbc7239998708674eb73460cfff49ba828b48e7787bc34155ec921f/mounts/shm/** --exclude .//var/lib/docker/containers/7eb12b88cdd9f3e09c21704b35632578624d6113fbc3450a176adc4ef1c0ad6c/mounts/shm/** --exclude .//var/lib/docker/containers/862970f34b4467fe5ca22ac63ffcd31d7a85f9685d32bf42cbbddffa766ae382/mounts/shm/** --exclude .//var/lib/kubelet/pods/fd858f1a-3cab-4215-accf-e8ac408a1767/volumes/kubernetes.io~secret/deepfence-agent-token-4nvfs/** --exclude .//var/lib/kubelet/pods/86432e1d-d38d-4ac6-82b0-9279cbe2d6da/volumes/kubernetes.io~secret/deepfence-agent-token-4nvfs/** --exclude .//var/lib/docker/containers/874ba3443ca148f296ec120c08cfceba98d6266a3b5dbf4a72b6d5e98fb23d89/mounts/shm/** --exclude .//var/lib/docker/containers/68b5613546c1fd7e585cc0a450402fc4c3a5f239a0048cb13d0d5cb47b76ba28/mounts/shm/** --exclude .//var/lib/docker/overlay2/5583bdf8b19a66d7058a2e93347134076081af6e43c32eebc67bbf964a0b1c9a/merged/dev/** --exclude .//var/lib/docker/overlay2/5583bdf8b19a66d7058a2e93347134076081af6e43c32eebc67bbf964a0b1c9a/merged/dev/shm/** --exclude .//var/lib/docker/overlay2/5583bdf8b19a66d7058a2e93347134076081af6e43c32eebc67bbf964a0b1c9a/merged/sys/fs/cgroup/** --exclude .//var/lib/docker/overlay2/5583bdf8b19a66d7058a2e93347134076081af6e43c32eebc67bbf964a0b1c9a/merged/run/containerd/containerd.sock/** --exclude ./run/secrets/kubernetes.io/serviceaccount/** --catalogers dpkgdb-cataloger --catalogers rpmdb-cataloger --catalogers apkdb-cataloger --catalogers alpmdb-cataloger --catalogers java-cataloger" time="2022-10-03 08:29:14" level=error msg="output: signal: killed" time="2022-10-03 08:29:17" level=error msg="error in generating sbom: signal: killed"

[ramanan-ravi]

Hi @huydq2vietcapitalbank, can you share the system configuration of the agent kubernetes cluster (cpu, memory)?

[shyam-dev]

Hello @huydq2vietcapitalbank - also share details about the cluster - version of kubernetes, managed kubernetes or built using kops/k3s etc..

[huydq2vietcapitalbank]

Hi @shyam-dev , I will update the information later, thank you.

Hi @ramanan-ravi, this is the system information of kubernetes cluster. CPU: 8 cores RAM: 8GB

And I found something in the agent log file.

<probe> ERRO: 2022/10/04 10:54:30.024281 docker registry: Get "http://unix.sock/containers/json?all=1": dial unix /var/run/docker.sock: connect: no such file or directory

<probe> WARN: 2022/10/05 02:02:20.350172 background /proc reader: full pass took 15.007917672s: 50% more than expected (10s) <probe> ERRO: 2022/10/05 02:02:23.936695 Error generating CRI report: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService <probe> ERRO: 2022/10/05 02:02:28.835982 Error generating CRI report: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

Deepfence Server log file (We use docker version) deepfenceio/deepfence_package_scanner_ce:1.4.1 [root@D-SecDeepfence ~]# docker logs 8eb1f66e4326 -f time="2022-10-04 09:17:30" level=info msg="trying to connect to endpoint 'unix:///var/run/docker.sock' with timeout '10s'" time="2022-10-04 09:17:30" level=warning msg="could not connect to endpoint 'unix:///var/run/docker.sock': dial unix /var/run/docker.sock: connect: no such file or directory" time="2022-10-04 09:17:30" level=info msg="trying to connect to endpoint 'unix:///run/containerd/containerd.sock' with timeout '10s'" time="2022-10-04 09:17:40" level=warning msg="could not connect to endpoint 'unix:///run/containerd/containerd.sock': context deadline exceeded" time="2022-10-04 09:17:40" level=info msg="trying to connect to endpoint 'unix:///run/k3s/containerd/containerd.sock' with timeout '10s'" time="2022-10-04 09:17:50" level=warning msg="could not connect to endpoint 'unix:///run/k3s/containerd/containerd.sock': context deadline exceeded" time="2022-10-04 09:17:50" level=error msg="Error detecting container runtime: could not detect container runtime" Starting server at port 8005

[ramanan-ravi]

@huydq2vietcapitalbank

error in generating sbom: signal: killed

This is possibly because of lack of memory in the k8s node. Can you check memory utilisation in the nodes?

kubectl top nodes

[wonhee0410]

@huydq2vietcapitalbank

error in generating sbom: signal: killed

This is possibly because of lack of memory in the k8s node. Can you check memory utilisation in the nodes?

kubectl top nodes

Hi, is there another way to fix this ? Because on another K8S with the same specs, it runs fine.

[ramanan-ravi]

Hi @wonhee0410 , please increase memory limit here, redeploy and try again

[wonhee0410]

Hi @wonhee0410 , please increase memory limit here, redeploy and try again

Hi, I'm using Threat Mapper docker and deploy Agent using Docker too. So how can I edit it ? Thanks.

[ramanan-ravi]

Hi @wonhee0410 , please increase memory limit here, redeploy and try again

Hi, I'm using Threat Mapper docker and deploy Agent using Docker too. So how can I edit it ? Thanks.

Depending on the number of files in the vm, scan may take 1gb - 2.5gb memory. Please ensure that during scan.

[wonhee0410]

Hi @wonhee0410 , please increase memory limit here, redeploy and try again

Hi, I'm using Threat Mapper docker and deploy Agent using Docker too. So how can I edit it ? Thanks.

Depending on the number of files in the vm, scan may take 1gb - 2.5gb memory. Please ensure that during scan.

I mean how can I edit the memory limit in Agent deployed by Docker.

[ramanan-ravi]

This docker run command does not set any memory limit. (only cpu limit is set). So if the vm has enough memory, it should work.

[wonhee0410]

This docker run command does not set any memory limit. (only cpu limit is set). So if the vm has enough memory, it should work.

Hi @ramanan-ravi , I have increased the memory of VM, but it still generates the same error.

time="2022-10-27 04:51:50" level=error msg="error from syft command for syftArgs: packages dir:/fenced/mnt/host/ -o json --file /tmp/0ZbQTRsQauLboutput.json -q --exclude ./var/lib/docker/** --exclude ./var/lib/containerd/** --exclude ./mnt/** --exclude ./run/** --exclude ./proc/** --exclude ./dev/** --exclude ./boot/** --exclude ./home/kubernetes/containerized_mounter/** --exclude ./sys/** --exclude ./lost+found/** --exclude ./dev/** --exclude ./sys/fs/cgroup/** --exclude ./dev/shm/** --exclude ./run/containerd/containerd.sock/** --exclude .//dev/shm/** --exclude .//sys/fs/cgroup/** --exclude .//run/** --exclude .//var/lib/docker/containers/af67116e6412fb6995ab1b5d91dc687e1ba033d515b540dd48b3234f094a7050/mounts/shm/** --exclude .//var/lib/kubelet/pods/baa9c25c-ed83-47a0-8fa1-963fabd2f764/volumes/kubernetes.io~secret/exporter-node-cluster-monitoring-token-zmptv/** --exclude .//var/lib/kubelet/pods/5a143f5d-776a-453f-9a29-f528f3c8b557/volumes/kubernetes.io~secret/cattle-token-jwxxw/** --exclude .//var/lib/kubelet/pods/a390022e-2907-44e4-aaef-f7a5aa52ad10/volumes/kubernetes.io~secret/default-token-zhjdd/** --exclude .//var/lib/kubelet/pods/5a143f5d-776a-453f-9a29-f528f3c8b557/volumes/kubernetes.io~secret/cattle-credentials/** --exclude .//var/lib/kubelet/pods/4990a854-0674-48be-8308-7f52c285697c/volumes/kubernetes.io~secret/kubernetes-dashboard-certs/** --exclude .//var/lib/kubelet/pods/180a7971-e365-4c46-ba46-5e8636dd8835/volumes/kubernetes.io~secret/cluster-monitoring-token-4cjgm/** --exclude .//var/lib/kubelet/pods/180a7971-e365-4c46-ba46-5e8636dd8835/volumes/kubernetes.io~secret/tls-assets/** --exclude .//var/lib/kubelet/pods/4990a854-0674-48be-8308-7f52c285697c/volumes/kubernetes.io~secret/kubernetes-dashboard-token-k4swc/** --exclude .//var/lib/kubelet/pods/9f4d3a78-2221-4df5-b2e9-11269c1bd146/volumes/kubernetes.io~secret/calico-node-token-2l52n/** --exclude .//var/lib/kubelet/pods/d88b93e9-24d1-4455-8586-dc6134f74d08/volumes/kubernetes.io~secret/kube-proxy-token-q868q/** --exclude .//var/lib/kubelet/pods/2f725d49-27cf-4a2a-91ed-77f23e07ef98/volumes/kubernetes.io~secret/default-token-zhjdd/** --exclude .//var/lib/kubelet/pods/180a7971-e365-4c46-ba46-5e8636dd8835/volumes/kubernetes.io~secret/config/** --exclude .//var/lib/docker/containers/c3ffcd9d95c6d4387b8a6822163be0f65956861fb0269a117376e1b6ed61091c/mounts/shm/** --exclude .//var/lib/docker/containers/2114ede5f8f69885d046d47e7b9da96e42a5fadf6c295f6f4f64e6b4c471db89/mounts/shm/** --exclude .//var/lib/docker/containers/4eda7482507b1c9a12324db5bc9a4d4627bcecdf55d44c179f7d6297e6e4b534/mounts/shm/** --exclude .//var/lib/docker/containers/1f32f51f1df1f86cc3a78ad9a82783f5772e6c5c19c88df4331cf182ea8f61d8/mounts/shm/** --exclude .//var/lib/docker/containers/4b5d41d8c8ccad82c642e38c336a8969904fb18a971949319106cb4ae3f8cc85/mounts/shm/** --exclude .//var/lib/docker/containers/ec2adc7c7264e39a3c890671ec3f6b4dfb9e7cc2f3d961867781095a53c73a31/mounts/shm/** --exclude .//var/lib/docker/containers/e932fae5fa960d0596534b0d7483bf5874c07f7174ed96e929fa834a55839284/mounts/shm/** --exclude .//var/lib/docker/containers/eedabe670c8f3a6ecc52d90c5ffdf4bfe6ee443612c865fb834a211cb0d7487e/mounts/shm/** --exclude .//var/lib/kubelet/pods/0c3f5336-09f6-4ddd-b925-81ba89cd4d18/volumes/kubernetes.io~secret/default-token-bfl5w/** --exclude .//var/lib/docker/containers/737b62a1fb42d93bbeb576d0f8ee61d520a00f11a42d00a805e6e8e127462674/mounts/shm/** --exclude .//var/lib/kubelet/pods/75015f5e-8a9f-4493-bd33-288692ec6ac3/volumes/kubernetes.io~secret/default-token-bfqr7/** --exclude .//var/lib/docker/containers/a0c789edd12ccd85fe2ba383630a0b4feab36a6ab7aaaefc15d34c16dffa523e/mounts/shm/** --exclude .//var/lib/kubelet/pods/e40e1906-4c57-43b8-8632-9e39e6d654c8/volumes/kubernetes.io~secret/default-token-bfqr7/** --exclude .//var/lib/docker/containers/1a6b52b0ce5a756ac081e9428fda41fc478f4f930eaa06598bc9ab4efe01b5e8/mounts/shm/** --exclude .//var/lib/kubelet/pods/eb1c6fa5-c6df-41fd-b0de-6c67bf8f8983/volumes/kubernetes.io~secret/default-token-bfqr7/** --exclude .//var/lib/docker/containers/8f21a5544ccef8d97bf7e9954c04febaf7214ff880967181704a4fbed3953560/mounts/shm/** --exclude .//var/lib/kubelet/pods/64877b2a-f59e-48f7-9b11-f6fc18f00a63/volumes/kubernetes.io~secret/default-token-bfqr7/** --exclude .//var/lib/docker/containers/dd4de73f1edbda92a1b64ff68c7c446e63f4798055378c411a4ba821006d2174/mounts/shm/** --exclude .//var/lib/kubelet/pods/8ef7decf-d1fe-42cc-8c7f-015c110de075/volumes/kubernetes.io~secret/default-token-bfqr7/** --exclude .//var/lib/docker/containers/f58cd998a5246a1aac7b2a8e3da941c9fc840c8a122c2830ac80e5a273697bc0/mounts/shm/** --exclude .//var/lib/kubelet/pods/cd040600-c75b-4ed5-92cc-a059e777dac0/volumes/kubernetes.io~secret/default-token-jsgn2/** --exclude .//var/lib/docker/containers/f1f732be4cd66535676b5f0b72c99c2a10615e91f7373221cdd6b87890b56992/mounts/shm/** --exclude .//var/lib/kubelet/pods/ee56a014-12e0-412a-baa7-db4b8d42a0d6/volumes/kubernetes.io~secret/default-token-jsgn2/** --exclude .//var/lib/docker/containers/5d59b3b75daeece08cff5397c49602b0b7105ba30b40d6b3b79fbdd2f38c3a43/mounts/shm/** --exclude .//var/lib/kubelet/pods/aac3c566-eaa0-4338-aded-8af7e996845a/volumes/kubernetes.io~secret/default-token-bfl5w/** --exclude .//var/lib/docker/containers/6a993b0c2c6ac9885edbbd53e5d6430ecb862696af173660847f4bad8962828e/mounts/shm/** --exclude .//var/lib/kubelet/pods/10978058-f148-4d52-aeee-ac2d51c54c63/volumes/kubernetes.io~secret/default-token-jsgn2/** --exclude .//var/lib/docker/containers/a34f4017fd4faf85cc8a9c5fd75db235bbed7658cc080347ac7ab46b507cecc6/mounts/shm/** --exclude .//var/lib/kubelet/pods/4ff441fb-9896-4eff-9429-6c7b3fbe493b/volumes/kubernetes.io~secret/default-token-pdsvn/** --exclude .//var/lib/docker/containers/ad1294070a7e346bbce589b7dc9bb17dec95900f3108566b0531258ed4bd3372/mounts/shm/** --exclude .//var/lib/kubelet/pods/f0cd368e-9319-46ba-819e-36089b8dfab4/volumes/kubernetes.io~secret/deepfence-agent-token-4nvfs/** --exclude .//var/lib/docker/containers/7be15e2a7429f2783ab2f35e75fec1eda46d9c31a55ca8d79e0d15441319e9bb/mounts/shm/** --exclude .//var/lib/docker/overlay2/10ca1ef580f0f39e38765ea3d20a567d1aa6596baf529c4841ec6a68c91d41d8/merged/dev/** --exclude .//var/lib/docker/overlay2/10ca1ef580f0f39e38765ea3d20a567d1aa6596baf529c4841ec6a68c91d41d8/merged/dev/shm/** --exclude .//var/lib/docker/overlay2/10ca1ef580f0f39e38765ea3d20a567d1aa6596baf529c4841ec6a68c91d41d8/merged/sys/fs/cgroup/** --exclude .//var/lib/docker/overlay2/10ca1ef580f0f39e38765ea3d20a567d1aa6596baf529c4841ec6a68c91d41d8/merged/run/containerd/containerd.sock/** --exclude ./run/secrets/kubernetes.io/serviceaccount/** --catalogers dpkgdb-cataloger --catalogers rpmdb-cataloger --catalogers apkdb-cataloger --catalogers alpmdb-cataloger --catalogers java-cataloger" time="2022-10-27 04:51:50" level=error msg="output: signal: killed" time="2022-10-27 04:51:54" level=error msg="error in generating sbom: signal: killed"

[wonhee0410]

Hi all,

We found another useful error log.

time="2022-10-13 07:14:54" level=info msg="trying to connect to endpoint 'unix:///var/run/docker.sock' with timeout '10s'" time="2022-10-13 07:14:54" level=warning msg="could not connect to endpoint 'unix:///var/run/docker.sock': dial unix /var/run/docker.sock: connect: no such file or directory" time="2022-10-13 07:14:54" level=info msg="trying to connect to endpoint 'unix:///run/containerd/containerd.sock' with timeout '10s'" time="2022-10-13 07:15:04" level=warning msg="could not connect to endpoint 'unix:///run/containerd/containerd.sock': context deadline exceeded" time="2022-10-13 07:15:04" level=info msg="trying to connect to endpoint 'unix:///run/k3s/containerd/containerd.sock' with timeout '10s'" time="2022-10-13 07:15:14" level=warning msg="could not connect to endpoint 'unix:///run/k3s/containerd/containerd.sock': context deadline exceeded" time="2022-10-13 07:15:14" level=error msg="Error detecting container runtime: could not detect container runtime" time="2022-10-13 07:15:14" level=info msg="main: server listening at /tmp/package-scanner.sock" time="2022-10-24 10:34:15" level=error msg="unable to get registry credentials"

<probe> ERRO: 2022/11/01 02:12:41.760404 docker registry: Get "http://unix.sock/containers/json?all=1": dial unix /var/run/docker.sock: connect: no such file or directory

[ibreakthecloud]

Hi @wonhee0410 Is this agent k8s or docker, and if it's k8s, what is the underlying container runtime?

[wonhee0410]

Hi @wonhee0410 Is this agent k8s or docker, and if it's k8s, what is the underlying container runtime?

Hi @ibreakthecloud ,

We're using docker agent.

[wonhee0410]

Now I still cannot use the Vulnerability scan on K8S server.