Docker Compose Deployment for securing unorchestrated container

[daemon1024]

Feature Request

Short Description

We currently have a systemd deployment which helps manage unorchestrated containers and host policies Ref https://github.com/kubearmor/KubeArmor/blob/main/getting-started/kubearmor_vm.md

Is your feature request related to a problem? Please describe the use case. Folks might want to just start another container and not deal with the package management hassle to start systemd service.

Describe the solution you'd like

Docker Compose File and Documentation to run KubeArmor directly with docker.

Here's how you can do it

docker run -v /opt/kubearmor/BPF:/opt/kubearmor/BPF --privileged kubearmor/kubearmor-init:stable

# Followed by

docker run -v /opt/kubearmor/BPF:/opt/kubearmor/BPF -v /sys/fs/bpf:/sys/fs/bpf -v /sys/fs/bpf:/sys/fs/bpf -v /sys/kernel/security:/sys/kernel/security -v /sys/kernel/debug:/sys/kernel/debug  -v /var/run/containerd/containerd.sock:/var/run/containerd/containerd.sock -v /run/containerd:/run/containerd -v /var/lib/docker:/var/lib/docker --privileged  --pid=host --ipc=host --net=host kubearmor/kubearmor:latest -k8s=false

This is privielged but we won't need the privileges and can mention exact capabilities as well in the docker compose file.

[yashvardhanmishra]

hey @daemon1024 . Could you pls assign this to me , i would like to work on this

[sarthaksarthak9]

I would love to solve this issue @daemon1024 Pls assign this issue to me

[yashvardhanmishra]

hey @daemon1024 can you please guide me related to the capabilities which is to mention ?

[sarthaksarthak9]

@yashvardhanmishra I am really sorry I by mistake link my pull request with other issue and you haven't get noticed and created pull request

[daemon1024]

Folks @yashvardhanmishra @sarthaksarthak9 Thanks a lot for the interest and raising the PRs already. Both of them look duplicated efforts, let's handle it in a single PR.

@sarthaksarthak9 why don't you help review @yashvardhanmishra PR since he wanted to work on the issue first. I appreciate both of your efforts a lot so thank you.

[sarthaksarthak9]

Folks @yashvardhanmishra @sarthaksarthak9 Thanks a lot for the interest and raising the PRs already. Both of them look duplicated efforts, let's handle it in a single PR.

@sarthaksarthak9 why don't you help review @yashvardhanmishra PR since he wanted to work on the issue first. I appreciate both of your efforts a lot so thank you.

yah sure why not

[daemon1024]

This is an open issue again

[KrishAryan]

I want to work on this issue. Please assign this issue to me

[navin772]

@daemon1024 I'm trying to solve this. The docker compose file works and starts kubearmor after kubearmor-init has finished running.

But the policy enforcement doesn't seem to work. For example, I created container for wordpress-mysql via the docker-compose file:

services:
  wordpress:
    container_name: wordpress-mysql
    image: wordpress:latest
    ports:
      - 80:80
    restart: always
    environment:
      - WORDPRESS_DB_HOST=db
      - WORDPRESS_DB_USER=wordpress
      - WORDPRESS_DB_PASSWORD=wordpress
      - WORDPRESS_DB_NAME=wordpress
volumes:
  db_data:

The creation of the container is successfully detected by KubeArmor (which is running as a docker container).

Then, I applied a block policy via karmor vm policy add ksp-wordpress-block-policy.yaml which is:

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-block-policy
spec:
  severity: 3
  selector:
    matchLabels:
      kubearmor.io/container.name: wordpress-mysql
  process:
    matchPaths:
    - path: /usr/bin/apt
    - path: /usr/bin/apt-get

      # apt update
      # apt-get update

  action:
    Block

The security policy is also detected by KubeArmor -

2024-05-24 17:31:01.656355      INFO    Started to monitor container security policies on gRPC
2024-05-24 17:31:01.656427      INFO    Started to serve gRPC-based log feeds
2024-05-24 17:31:01.656437      INFO    Initialized KubeArmor
2024-05-24 17:31:01.656473      WARN    Policies dir not found for restoration


2024-05-24 17:31:16.784291      INFO    Detected a Container Security Policy (added/container_namespace/ksp-block-policy)

But now if I exec into the wordpress-mysql container and run apt update it isn't blocking that!

[navin772]

@DelusionalOptimist I want to work on this issue, please assign me.

[KrishAryan]

you can work on this issue, my friend

[DelusionalOptimist]

@navin772 can you check the output of cat /sys/kernel/security/lsm? If it doesn't have bpf in it, check this out - https://github.com/kubearmor/KubeArmor/wiki/Support-for-non-orchestrated-containers#policy-enforcement-for-containers-running-in-docker-with-apparmor

[navin772]

@DelusionalOptimist I have bpf as the lsm:

$ cat /sys/kernel/security/lsm
lockdown,capability,landlock,yama,apparmor,bpf

I have tried KubeArmor in systemd mode and enforcement works, the only difference I see is that the logs of kubearmor.service shows that it is using docker.sock :

May 28 12:39:44 navin kubearmor[2817]: 2024-05-28 12:39:44.521603        INFO        Detected a container (added/0ed465588467)
May 28 12:39:44 navin kubearmor[2817]: 2024-05-28 12:39:44.521623        INFO        Using unix:///var/run/docker.sock for monitoring containers
May 28 12:39:44 navin kubearmor[2817]: 2024-05-28 12:39:44.521641        INFO        Started to monitor Docker events

and when running kubearmor as a docker container, it uses containerd.sock:

kubearmor-1       | 2024-05-28 12:44:04.035532  INFO    Starting TraceEvents from BPF LSM Enforcer
kubearmor-1       | 2024-05-28 12:44:04.035608  INFO    Using unix:///run/containerd/containerd.sock for monitoring containers
kubearmor-1       | 2024-05-28 12:44:04.036001  INFO    Initialized Containerd Handler
kubearmor-1       | 2024-05-28 12:54:21.926524  INFO    Successfully added visibility map with key={PidNS:4026532490 MntNS:4026532487} to the kernel
kubearmor-1       | 2024-05-28 12:54:21.926620  INFO    Detected a container (added/c2f56b86c255/pidns=4026532490/mntns=4026532487)
kubearmor-1       | 2024-05-28 12:55:02.510209  INFO    Detected a Container Security Policy (added/container_namespace/ksp-block-policy)

Can that be the issue? Although container and policy detection seems to work.

[navin772]

containerd runtime is currently not supported as mentioned in https://github.com/kubearmor/KubeArmor/issues/1426. Enforcement with docker runtime works.

Commands:

1. docker run -v /tmp/:/opt/kubearmor/BPF kubearmor/kubearmor-init:stable
2. docker run -v /tmp/:/opt/kubearmor/BPF -v /sys/fs/bpf:/sys/fs/bpf -v /sys/kernel/security:/sys/kernel/security -v /sys/kernel/debug:/sys/kernel/debug -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --privileged --pid=host --ipc=host --net=host kubearmor/kubearmor:latest -k8s=false

--privileged flag is not required for kubearmor-init when using the /tmp/ dir but in the kubearmor container it is required (even if we explicitly list the capabilities).

[DelusionalOptimist]

@navin772 Do we get any specific errors while running with explicitly listed capabilities or it's just that enforcement doesn't work? :eyes:

[navin772]

@DelusionalOptimist kubearmor fails to start with error:

kubearmor-1       | 2024-06-21 10:05:50.066540	INFO	Node Name:
kubearmor-1       | 2024-06-21 10:05:50.066545	INFO	Node IP:
kubearmor-1       | 2024-06-21 10:05:50.066550	INFO	OS Image:
kubearmor-1       | 2024-06-21 10:05:50.066554	INFO	Kernel Version:
kubearmor-1       | 2024-06-21 10:05:50.066807	INFO	Initialized KubeArmor Logger
kubearmor-1       | 2024-06-21 10:05:50.067695	INFO	Detected mounted BPF filesystem at /sys/fs/bpf
kubearmor-1       | 2024-06-21 10:05:50.067968	INFO	Initializing eBPF system monitor
kubearmor-1       | panic: runtime error: invalid memory address or nil pointer dereference
kubearmor-1       | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x193ebc6]
kubearmor-1       |
kubearmor-1       | goroutine 1 [running]:
kubearmor-1       | github.com/cilium/ebpf.(*Map).Update(0x1d3ed20?, {0x1d3ed20?, 0xc0000e3e80?}, {0x1f2aa80?, 0xc000182280?}, 0xc00030f1f8?)
kubearmor-1       | 	/go/pkg/mod/github.com/cilium/[email protected]/map.go:724 +0x26
kubearmor-1       | github.com/cilium/ebpf.(*Map).Put(...)
kubearmor-1       | 	/go/pkg/mod/github.com/cilium/[email protected]/map.go:719
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/monitor.(*SystemMonitor).UpdateNsKeyMap(0xc0001fcc00, {0x1f69ac7?, 0x1f74be5?}, {0x0, 0x0}, {0x12?, 0x2b?, 0x55?, 0x0?})
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/monitor/systemMonitor.go:343 +0x52e
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/monitor.(*SystemMonitor).UpdateVisibility(0xc0001fcc00)
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/monitor/systemMonitor.go:435 +0x22a
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/monitor.(*SystemMonitor).initBPFMaps(0xc0001fcc00)
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/monitor/systemMonitor.go:228 +0x116
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/monitor.(*SystemMonitor).InitBPF(0xc0001fcc00)
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/monitor/systemMonitor.go:470 +0x1f8
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/core.(*KubeArmorDaemon).InitSystemMonitor(0xc000525800)
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/core/kubeArmor.go:257 +0x8a
kubearmor-1       | github.com/kubearmor/KubeArmor/KubeArmor/core.KubeArmor()
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/core/kubeArmor.go:533 +0xf56
kubearmor-1       | main.main()
kubearmor-1       | 	/usr/src/KubeArmor/KubeArmor/main.go:79 +0x3ed
kubearmor-1 exited with code 2