KubeArmor
KubeArmor copied to clipboard
kubearmor
Minikube Start is failing with latest version of minikube (1.26) and k8s(1.24)
Minikube start
General Information
- Environment (minikube)
- Kernel version (Linux eswar-LEGION 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux)
To Reproduce
- Goto contributions minikube.
- Install minikube and virtualbox using scripts mentioned under contributions/minikube dir.
- Then run minikube with start_minikube.sh script.
Observed behavior
Minikube throws an error and fails to start.
Expected behavior Minikube is expected to start with no issues/errors.
Temporary solution
with minikube start, use the below flag
--kubernetes-version=1.23.1
This seems interesting! I would love to help out in this @seswarrajan :)
I am new to the KubeArmor community & it would be really great if you could guide me a bit or provide some reference pointers on how to get started Thanks!
Would try to reproduce the issue for starters!
Hello @verma-kunal Great Day. Happy to hear your interest in this.
I will assign the issue to you.
An update on this! Was a bit caught up in some personal stuff & will be resuming work on this. Will update soon with the progress
I tried using latest version of minikube with virtualbox driver. I was able to get past initial iso installation and the VM booted up fine. The problem was kubearmor was not starting up due to volume mount paths issue. Also minikube VM does not seem to have certain file paths (/sys/kernel/security/lsms) that kubearmor expects to have.
kubectl get pods status
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d4b75cb6d-fsg6n 1/1 Running 0 57m
kube-system etcd-minikube 1/1 Running 0 57m
kube-system kube-apiserver-minikube 1/1 Running 0 57m
kube-system kube-controller-manager-minikube 1/1 Running 0 57m
kube-system kube-proxy-4tz7h 1/1 Running 0 57m
kube-system kube-scheduler-minikube 1/1 Running 0 57m
kube-system kubearmor-annotation-manager-85857fc8d7-q2xc9 0/2 ContainerCreating 0 52m
kube-system kubearmor-host-policy-manager-84d88d59b7-hpdvp 2/2 Running 0 52m
kube-system kubearmor-policy-manager-64489f77d4-xp9br 2/2 Running 0 52m
kube-system kubearmor-relay-64c6fff875-ps2k7 1/1 Running 0 52m
kube-system kubearmor-tp85j 0/1 ContainerCreating 0 52m
kube-system storage-provisioner 1/1 Running 2 (4m ago) 57m
kubectl describe of kubearmor pod
❯ k describe pod -n kube-system kubearmor-tp85j
Name: kubearmor-tp85j
Namespace: kube-system
Priority: 0
Node: minikube/192.168.59.100
Start Time: Thu, 28 Jul 2022 00:57:09 +0530
Labels: controller-revision-hash=c94c94765
kubearmor-app=kubearmor
pod-template-generation=1
Annotations: container.apparmor.security.beta.kubernetes.io/kubearmor: unconfined
Status: Pending
IP: 192.168.59.100
IPs:
IP: 192.168.59.100
Controlled By: DaemonSet/kubearmor
Containers:
kubearmor:
Container ID:
Image: kubearmor/kubearmor:stable
Image ID:
Port: 32767/TCP
Host Port: 32767/TCP
Args:
-gRPC=32767
-logPath=/tmp/kubearmor.log
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: exec [/bin/bash -c if [ -z $(pgrep kubearmor) ]; then exit 1; fi;] delay=60s timeout=1s period=10s #success=1 #failure=3
Environment:
KUBEARMOR_NODENAME: (v1:spec.nodeName)
Mounts:
/etc/apparmor.d from etc-apparmor-d-path (rw)
/lib/modules from lib-modules-path (ro)
/media/root/etc/os-release from os-release-path (ro)
/sys/fs/bpf from sys-fs-bpf-path (rw)
/sys/kernel/debug from sys-kernel-debug-path (rw)
/sys/kernel/security from sys-kernel-security-path (rw)
/usr/src from usr-src-path (ro)
/var/lib/docker from docker-storage-path (ro)
/var/run/docker.sock from docker-sock-path (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-svztq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules-path:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType: Directory
sys-fs-bpf-path:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: Directory
sys-kernel-security-path:
Type: HostPath (bare host directory volume)
Path: /sys/kernel/security
HostPathType: Directory
sys-kernel-debug-path:
Type: HostPath (bare host directory volume)
Path: /sys/kernel/debug
HostPathType: Directory
os-release-path:
Type: HostPath (bare host directory volume)
Path: /etc/os-release
HostPathType: File
usr-src-path:
Type: HostPath (bare host directory volume)
Path: /usr/src
HostPathType: Directory
etc-apparmor-d-path:
Type: HostPath (bare host directory volume)
Path: /etc/apparmor.d
HostPathType: DirectoryOrCreate
docker-sock-path:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType: Socket
docker-storage-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker
HostPathType: DirectoryOrCreate
kube-api-access-svztq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 47m default-scheduler Successfully assigned kube-system/kubearmor-tp85j to minikube
Warning FailedMount 45m kubelet Unable to attach or mount volumes: unmounted volumes=[sys-kernel-security-path usr-src-path], unattached volumes=[etc-apparmor-d-path sys-fs-bpf-path sys-kernel-security-path os-release-path docker-storage-path kube-api-access-svztq lib-modules-path usr-src-path sys-kernel-debug-path docker-sock-path]: timed out waiting for the condition
Warning FailedMount 42m kubelet Unable to attach or mount volumes: unmounted volumes=[usr-src-path sys-kernel-security-path], unattached volumes=[usr-src-path kube-api-access-svztq sys-kernel-security-path sys-kernel-debug-path os-release-path lib-modules-path sys-fs-bpf-path etc-apparmor-d-path docker-sock-path docker-storage-path]: timed out waiting for the condition
Warning FailedMount 40m (x11 over 47m) kubelet MountVolume.SetUp failed for volume "sys-kernel-security-path" : hostPath type check failed: /sys/kernel/security is not a directory
Warning FailedMount 40m kubelet Unable to attach or mount volumes: unmounted volumes=[usr-src-path sys-kernel-security-path], unattached volumes=[sys-kernel-debug-path docker-storage-path kube-api-access-svztq usr-src-path etc-apparmor-d-path os-release-path sys-fs-bpf-path sys-kernel-security-path docker-sock-path lib-modules-path]: timed out waiting for the condition
Warning FailedMount 31m kubelet Unable to attach or mount volumes: unmounted volumes=[sys-kernel-security-path usr-src-path], unattached volumes=[lib-modules-path os-release-path docker-sock-path sys-kernel-security-path sys-kernel-debug-path kube-api-access-svztq sys-fs-bpf-path usr-src-path etc-apparmor-d-path docker-storage-path]: timed out waiting for the condition
Warning FailedMount 16m (x23 over 47m) kubelet MountVolume.SetUp failed for volume "usr-src-path" : hostPath type check failed: /usr/src is not a directory
Warning FailedMount 117s (x13 over 29m) kubelet (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[usr-src-path sys-kernel-security-path], unattached volumes=[usr-src-path docker-sock-path lib-modules-path kube-api-access-svztq sys-kernel-security-path sys-kernel-debug-path os-release-path etc-apparmor-d-path sys-fs-bpf-path docker-storage-path]: timed out waiting for the condition
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565
hello, I am new to KubeArmor, I'm not sure if my problem is related to this issue, I am installing KubeArmor on minikube but it is taking a long time to run a pod and is showing
with latest minikube
kube-system kubearmor-vwrgl 0/1 Init:CrashLoopBackOff 6 (4m30s ago) 11m
Hey @verma-kunal , any update on this? If you need any help, please ping on the kubearmor slack channels. This fix is intended to be shipped as part of v0.7 release.
Hey @verma-kunal , any update on this? If you need any help, please ping on the kubearmor slack channels. This fix is intended to be shipped as part of v0.7 release.
Hey @nyrahul , actually I had been caught with some work here so couldn't work on this issue! Really sorry for not informing early on. I understand this has to be a part of v0.7 release & would start working on this asap
Hey @nyrahul , sadly I am not actually able to work on this issue due to prior commitments at the moment! I believe it would be best if another contributor could be assigned here, so that this isn't stalled further.
I hope you understand! Thanks
Hey @nyrahul , sadly I am not actually able to work on this issue due to prior commitments at the moment! I believe it would be best if another contributor could be assigned here, so that this isn't stalled further.
I hope you understand! Thanks
No problem. Thanks for informing
Please I would like to work on this issue
Thank you for the interest. As you can see, this issue is in v0.7 release plan and thus needs relatively faster resolution. Please let us know on slack/here if you have any questions.
Thanks since it works with the flag --kubernetes-version=1.23.1. How bout we just add it to the minikube start script
Thanks since it works with the flag --kubernetes-version=1.23.1. How bout we just add it to the minikube start script
Lots of folks use kubearmor on minikube without using our start script. Thus if someone installs minikube from scratch and uses kubearmor on it, it should work as well. Secondly, keeping attached to an old version is not a good idea.
Thanks a lot for the opportunity to contribute to this issue. Researching on this issue. I found out this is caused because minikube by defaults starts with the latest version of Kubernetes which turns out have dropped support for docker. Users would have to use an older version of kubernetes if they would still like to use dockershim or install cri-docker runtime. Link to related issue on minikube: https://github.com/kubernetes/minikube/issues/14410 Please let me know if I am on the right path
@Chinwendu20 Can we use a different container runtime in that case? containerd maybe?
I am unsure if this is something that we would have to do from our end. It seems like it is something users that decide to use minikube would have to do for themselves.
Hello! I have not been able to get your thoughts on this. Anytime is fine though
Hey @Chinwendu20 thanks for taking a look at it. We currently recommend minikube to be setup using our script https://github.com/kubearmor/KubeArmor/blob/main/contribution/minikube/start_minikube.sh, you can add --container-runtime= containerd flag with the minikube start command and check if that works. If yes, then we can update the script and the documentation to mention about using containerd runtime.
Thanks again for trusting me with the issue as well as for your guidance. It seems there is a problem with the minikube recompiled image by Kubeamor. We need to bump its containerd version. As I am getting this error:
I think in general we should use the latest minikube iso image and then recompile it, as modifications have been made to the image to accommodate the change caused by Kubernetes 1.24. I tried using it and it works without any extra --kubernetes-version and ---container-runtime flag What do you think?
I think in general we should use the latest minikube iso image and then recompile it
:+1: I think it makes sense to use the latest version of minikube. cc: @daemon1024
I would suggest to approach the issue with a fresh angle. We shouldn't ideally be needing a custom Minikube ISO at all now! So no need to recompile things.
I think we should directly try KubeArmor on a fresh minikube cluster, then debug and check how we can make KubeArmor work their.
cc @Ankurk99 @Chinwendu20
Thanks @daemon1024. As per this documentation:
https://github.com/kubearmor/KubeArmor/tree/main/contribution/minikube
The issue with running kubearmor on minikube is that apparmor is not enabled by default and it does not also ship with ebpf capabilities.
That has still not changed as per this open issue on minikube:
https://github.com/kubernetes/minikube/issues/8299
I also ran it on my minikube instance:
Expected outcome:
Reference: https://kubernetes.io/docs/tutorials/security/apparmor/
If we do not use a custom image, we could add lines to automate enabling AppArmor and ebpf in the startup_minikube.sh. Following these steps:
For apparmor: https://wiki.archlinux.org/title/AppArmor
For ebpf: https://minikube.sigs.k8s.io/docs/tutorials/ebpf_tools_in_minikube/
As far as I understand. What do you think?
@Chinwendu20 Can you share output for
cat /boot/config-$(uname -r) | grep 'BPF'
and
cat /sys/kernel/security/lsm
on the minikube node?
I checked out the source code, https://github.com/kubernetes/minikube/blob/aca5f29b40319bf79de1e3eb15edcca7da7b5481/deploy/iso/minikube-iso/board/minikube/x86_64/linux_x86_64_defconfig#L44
I see that BPF is now enabled in the ISO. The documentation seems to be old since the new image is using 5.10 but performance image is kernel 4.19.
What happens when you try to install KubeArmor in the Minikube setup you have?
Hi @Chinwendu20, hope you are doing good. Wanted to ask if you are working on this issue?
Hello, @Ankurk99 thanks for checking up. Sorry that I have not been in touch. However, in response to @daemon1024 question., I tagged him to the answer on Slack
Hello, @Ankurk99 thanks for checking up. Sorry that I have not been in touch. However, in response to @daemon1024 question., I tagged him to an answer on Slack
Thanks for assigning this issue to me. I have gone through it. From what I understand it is about finding an alternate approach to check for LSMs. From what I understand our current issue is that minikube's image does not come with apparmor enabled. I do not think that would change even if we find an alternate means of checking for LSMs.
Cc: @daemon1024