Un-orchestrated KubeArmor: No telemetry when using containerd as a runtime

[Ankurk99]

Bug Report

General Information Mode of running: Containerized mode (-k8s=false) Container runtime: containerd

Description KubeArmor when running as a containerized application (unorchestrated) doesn't show any telemetry data. i.e. karmor logs --logFilter all is empty.

To Reproduce

1. Run KubeArmor in containerized mode

docker run -v /tmp/:/opt/kubearmor/BPF --privileged kubearmor/kubearmor-init:latest
docker run -d -v /tmp/:/opt/kubearmor/BPF -v /sys/fs/bpf:/sys/fs/bpf -v /sys/kernel/security:/sys/kernel/security -v /sys/kernel/debug:/sys/kernel/debug -v /etc/apparmor.d:/etc/apparmor.d -v /var/run/containerd/containerd.sock:/var/run/containerd/containerd.sock -v /run/containerd:/run/containerd -v /var/lib/docker:/var/lib/docker --privileged  --pid=host --ipc=host kubearmor/kubearmor:latest -k8s=false

2. Run some commands in another test container
3. Check the telemetry using karmor logs --logFilter all --gRPC=localhost:32767

Expected behavior

Telemetry data should be shown in the karmor logs output.

Note The telemetry functions properly when using the docker container runtime. The following command, when using the docker runtime, will display telemetry:

docker run -v /tmp/:/opt/kubearmor/BPF --privileged kubearmor/kubearmor-init:latest
docker run -d -v /tmp/:/opt/kubearmor/BPF -v /sys/fs/bpf:/sys/fs/bpf -v /sys/kernel/security:/sys/kernel/security -v /sys/kernel/debug:/sys/kernel/debug -v /etc/apparmor.d:/etc/apparmor.d -v /var/run/docker.sock:/var/run/docker.sock -v /run/docker:/run/docker -v /var/lib/docker:/var/lib/docker --privileged  --pid=host --ipc=host kubearmor/kubearmor:latest -k8s=false

[pavan12395]

Hey @Ankurk99 , I want to work on this issue , can you please assign this to me

[ShubhamTatvamasi]

@pavan12395 any update on this?

[sheharyaar]

is anyone working on this ?

[sheharyaar]

@Ankurk99 , I tried debugging this. Here is the log from docker as the container runtime :

karmor version 0.14.2 linux/amd64 BuildDate=2023-10-25T13:36:21Z kubearmor v1.0.3

2023-11-28 09:13:56.215239      WARN    Error creating kubernetes config, invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2023-11-28 09:13:56.215345      INFO    BUILD-INFO: commit: aba848a, branch: main, date: 2023-11-28T06:56:24Z
2023-11-28 09:13:56.215400      INFO    Arguments [bpfFsPath:/sys/fs/bpf cluster:default coverageTest:false criSocket: debug:false defaultCapabilitiesPosture:audit defaultFilePosture:audit defaultNetworkPosture:audit enableKubeArmorHostPolicy:false enableKubeArmorPolicy:true enableKubeArmorVm:false enforcerAlerts:true gRPC:32767 host:404c7c9dc05c hostDefaultCapabilitiesPosture:audit hostDefaultFilePosture:audit hostDefaultNetworkPosture:audit hostVisibility:default k8s:true kubeconfig: logPath:none lsm:bpf,apparmor,selinux seLinuxProfileDir:/tmp/kubearmor.selinux untrackedNs:kube-system,kubearmor visibility:process,file,network,capabilities]
2023-11-28 09:13:56.215481      INFO    Configuration [{Cluster:default Host:404c7c9dc05c GRPC:32767 LogPath:none SELinuxProfileDir: CRISocket: Visibility:process,file,network,capabilities HostVisibility:default Policy:true HostPolicy:false KVMAgent:false K8sEnv:false Debug:false DefaultFilePosture:audit DefaultNetworkPosture:audit DefaultCapabilitiesPosture:audit HostDefaultFilePosture:audit HostDefaultNetworkPosture:audit HostDefaultCapabilitiesPosture:audit CoverageTest:false ConfigUntrackedNs:[] LsmOrder:[] BPFFsPath: EnforcerAlerts:false}]
2023-11-28 09:13:56.215517      INFO    Final Configuration [{Cluster:default Host:404c7c9dc05c GRPC:32767 LogPath:none SELinuxProfileDir: CRISocket: Visibility:process,file,network,capabilities HostVisibility:none Policy:true HostPolicy:false KVMAgent:false K8sEnv:false Debug:false DefaultFilePosture:audit DefaultNetworkPosture:audit DefaultCapabilitiesPosture:audit HostDefaultFilePosture:audit HostDefaultNetworkPosture:audit HostDefaultCapabilitiesPosture:audit CoverageTest:false ConfigUntrackedNs:[kube-system kubearmor] LsmOrder:[bpf apparmor selinux] BPFFsPath:/sys/fs/bpf EnforcerAlerts:true}]
2023-11-28 09:13:56.215532      INFO    Node Name: 
2023-11-28 09:13:56.215542      INFO    Node IP: 
2023-11-28 09:13:56.215550      INFO    OS Image: 
2023-11-28 09:13:56.215558      INFO    Kernel Version: 
2023-11-28 09:13:56.215692      INFO    Initialized KubeArmor Logger
2023-11-28 09:13:56.216494      INFO    Detected mounted BPF filesystem at /sys/fs/bpf
2023-11-28 09:13:56.216659      INFO    Initializing eBPF system monitor
2023-11-28 09:13:56.236114      INFO    Successfully added visibility map with key={PidNS:0 MntNS:0} to the kernel
2023-11-28 09:13:56.266073      INFO    Successfully added visibility map with key={PidNS:12648430 MntNS:12648430} to the kernel
2023-11-28 09:13:56.266168      INFO    eBPF system monitor object file path: /opt/kubearmor/BPF/system_monitor.bpf.o
2023-11-28 09:13:56.705630      INFO    Initialized the eBPF system monitor
2023-11-28 09:13:56.939552      INFO    Initialized KubeArmor Monitor
2023-11-28 09:13:56.939687      INFO    Started to monitor system events
2023-11-28 09:13:56.941160      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:56.941615      INFO    Supported LSMs: capability,landlock,lockdown,yama,bpf
2023-11-28 09:13:59.702090      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:59.715223      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:59.717746      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:59.717759      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:59.746734      DEBUG   Pushing Telemetry without source
2023-11-28 09:13:59.750274      DEBUG   Pushing Telemetry without source
2023-11-28 09:14:03.547567      INFO    Initialized BPF-LSM Enforcer
2023-11-28 09:14:03.547600      INFO    Initialized KubeArmor Enforcer
2023-11-28 09:14:03.547606      INFO    Started to protect containers
2023-11-28 09:14:03.547637      INFO    Namespace container_namespace visibiliy configured {File:true Process:true Network:true Capabilities:true}
2023-11-28 09:14:03.547640      INFO    Starting TraceEvents from BPF LSM Enforcer
2023-11-28 09:14:03.553239      INFO    Verifying Docker API client version: 1.43
2023-11-28 09:14:03.557531      INFO    Initialized Docker Handler (version: 1.43)
2023-11-28 09:14:03.589443      INFO    Successfully added visibility map with key={PidNS:4026531836 MntNS:4026532973} to the kernel
2023-11-28 09:14:03.589520      INFO    Detected a container (added/404c7c9dc05c)
2023-11-28 09:14:03.589549      INFO    Using unix:///var/run/docker.sock for monitoring containers
2023-11-28 09:14:03.589599      INFO    Started to monitor Docker events
2023-11-28 09:14:04.590498      INFO    Started to monitor container security policies on gRPC
2023-11-28 09:14:04.590621      INFO    Started to serve gRPC-based log feeds
2023-11-28 09:14:04.590644      INFO    Initialized KubeArmor
2023-11-28 09:14:04.590728      WARN    Policies dir not found for restoration

when I execute karmor logs, I get the following errors :

$ karmor logs
Error: Get "https://192.168.39.197:8443/api/v1/pods?labelSelector=kubearmor-app%3Dkubearmor-relay": dial tcp 192.168.39.197:8443: connect: no route to host
$ karmor logs --logFilter all --gRPC=localhost:32767
Error: unable to create log client

I used the steps as mention by you in the original comment.