Error in run: file node.yaml not found for version cis-1.23

[Algoss]

Overview I have config file in /etc/kubernetes-kubelet/kubelet_config.yaml path. I added /etc/kubernetes-kubelet/kubelet_config.yaml to config.yaml and mounted them via configmap to pod in the path /opt/kube-bench/cfg.

While starting kube-bench pod in k8s cluster, it gives error Error in run: file node.yaml not found for version cis-1.23

How did you run kube-bench?

I ran kube-bench pod in k8s cluster. kubectl apply -f kube-bench-pod.yaml

What happened?

I1118 14:22:54.676795 15498 util.go:486] Checking for oc I1118 14:22:54.676867 15498 util.go:515] Can't find oc command: exec: "oc": executable file not found in $PATH I1118 14:22:54.676879 15498 kubernetes_version.go:36] Try to get version from Rest API I1118 14:22:54.676931 15498 kubernetes_version.go:161] Loading CA certificate I1118 14:22:54.678044 15498 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version I1118 14:22:54.683608 15498 kubernetes_version.go:100] vd: { "major": "1", "minor": "19", "gitVersion": "v1.19.16", "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074", "gitTreeState": "clean", "buildDate": "2021-10-27T16:20:18Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" } I1118 14:22:54.683663 15498 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} I1118 14:22:54.683676 15498 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16} I1118 14:22:54.683708 15498 kubernetes_version.go:36] Try to get version from Rest API I1118 14:22:54.683753 15498 kubernetes_version.go:161] Loading CA certificate I1118 14:22:54.683768 15498 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version I1118 14:22:54.688118 15498 kubernetes_version.go:100] vd: { "major": "1", "minor": "19", "gitVersion": "v1.19.16", "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074", "gitTreeState": "clean", "buildDate": "2021-10-27T16:20:18Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" } I1118 14:22:54.688159 15498 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} I1118 14:22:54.688172 15498 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16} I1118 14:22:54.688212 15498 common.go:281] mapToBenchmarkVersion for k8sVersion: "1.19" cisVersion: "cis-1.23" found: true I1118 14:22:54.688223 15498 common.go:347] Mapped Kubernetes version: 1.19 to Benchmark version: cis-1.23 I1118 14:22:54.688231 15498 common.go:350] Kubernetes version: "1.19" to Benchmark version: "cis-1.23" I1118 14:22:54.688238 15498 run.go:40] Checking targets [node] for cis-1.23 I1118 14:22:54.688341 15498 common.go:267] No version-specific config.yaml file in cfg/cis-1.23 I1118 14:22:54.688350 15498 common.go:273] Using config file: cfg/cis-1.23/config.yaml

What did you expect to happen:

I expected kube-bench pod to start successfully and scan the k8s cluster with correct config file /etc/kubernetes-kubelet/kubelet_config.yaml

Environment

Kube-bench: 0.6.8 Kuberneted: 1.19.16

Running processes I am running kube-bench pod in k8s cluster.

ConfigMap: config.yaml

`--- master: components: - apiserver - scheduler - controllermanager - etcd - flanneld # kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark - kubernetes - kubelet

kubernetes: defaultconf: /etc/kubernetes/config

apiserver: bins: - "kube-apiserver" - "hyperkube apiserver" - "hyperkube kube-apiserver" - "apiserver" - "openshift start master api" - "hypershift openshift-kube-apiserver" confs: - /etc/kubernetes/manifests/kube-apiserver.yaml - /etc/kubernetes/manifests/kube-apiserver.yml - /etc/kubernetes/manifests/kube-apiserver.manifest - /var/snap/kube-apiserver/current/args - /var/snap/microk8s/current/args/kube-apiserver - /etc/origin/master/master-config.yaml - /etc/kubernetes/manifests/talos-kube-apiserver.yaml defaultconf: /etc/kubernetes/manifests/kube-apiserver.yaml

scheduler: bins: - "kube-scheduler" - "hyperkube scheduler" - "hyperkube kube-scheduler" - "scheduler" - "openshift start master controllers" confs: - /etc/kubernetes/manifests/kube-scheduler.yaml - /etc/kubernetes/manifests/kube-scheduler.yml - /etc/kubernetes/manifests/kube-scheduler.manifest - /var/snap/kube-scheduler/current/args - /var/snap/microk8s/current/args/kube-scheduler - /etc/origin/master/scheduler.json - /etc/kubernetes/manifests/talos-kube-scheduler.yaml defaultconf: /etc/kubernetes/manifests/kube-scheduler.yaml kubeconfig: - /etc/kubernetes/scheduler.conf - /var/lib/kube-scheduler/kubeconfig - /var/lib/kube-scheduler/config.yaml - /system/secrets/kubernetes/kube-scheduler/kubeconfig defaultkubeconfig: /etc/kubernetes/scheduler.conf

controllermanager: bins: - "kube-controller-manager" - "kube-controller" - "hyperkube controller-manager" - "hyperkube kube-controller-manager" - "controller-manager" - "openshift start master controllers" - "hypershift openshift-controller-manager" confs: - /etc/kubernetes/manifests/kube-controller-manager.yaml - /etc/kubernetes/manifests/kube-controller-manager.yml - /etc/kubernetes/manifests/kube-controller-manager.manifest - /var/snap/kube-controller-manager/current/args - /var/snap/microk8s/current/args/kube-controller-manager - /etc/kubernetes/manifests/talos-kube-controller-manager.yaml defaultconf: /etc/kubernetes/manifests/kube-controller-manager.yaml kubeconfig: - /etc/kubernetes/controller-manager.conf - /var/lib/kube-controller-manager/kubeconfig - /system/secrets/kubernetes/kube-controller-manager/kubeconfig defaultkubeconfig: /etc/kubernetes/controller-manager.conf

etcd: optional: true bins: - "etcd" - "openshift start etcd" confs: - /etc/kubernetes/manifests/etcd.yaml - /etc/kubernetes/manifests/etcd.yml - /etc/kubernetes/manifests/etcd.manifest - /etc/etcd/etcd.conf - /var/snap/etcd/common/etcd.conf.yml - /var/snap/etcd/common/etcd.conf.yaml - /var/snap/microk8s/current/args/etcd - /usr/lib/systemd/system/etcd.service defaultconf: /etc/kubernetes/manifests/etcd.yaml

flanneld: optional: true bins: - flanneld defaultconf: /etc/sysconfig/flanneld

kubelet: optional: true bins: - "hyperkube kubelet" - "kubelet"

node: components: - kubelet - proxy # kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark - kubernetes

kubernetes: defaultconf: "/etc/kubernetes/config"

kubelet: cafile: - "/etc/kubernetes/pki/ca.crt" - "/etc/kubernetes/certs/ca.crt" - "/etc/kubernetes/cert/ca.pem" - "/var/snap/microk8s/current/certs/ca.crt" svc: # These paths must also be included # in the 'confs' property below - "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" - "/etc/systemd/system/kubelet.service" - "/lib/systemd/system/kubelet.service" - "/etc/systemd/system/snap.kubelet.daemon.service" - "/etc/systemd/system/snap.microk8s.daemon-kubelet.service" - "/etc/systemd/system/atomic-openshift-node.service" - "/etc/systemd/system/origin-node.service" bins: - "hyperkube kubelet" - "kubelet" kubeconfig: - "/etc/kubernetes/kubelet.conf" - "/etc/kubernetes/kubelet-kubeconfig.conf" - "/var/lib/kubelet/kubeconfig" - "/etc/kubernetes/kubelet-kubeconfig" - "/etc/kubernetes/kubelet/kubeconfig" - "/var/snap/microk8s/current/credentials/kubelet.config" - "/etc/kubernetes/kubeconfig-kubelet" confs: - "/etc/kubernetes-kubelet/kubelet_config.yaml" - "/etc/kubernetes/kubelet-config.yaml" - "/var/lib/kubelet/config.yaml" - "/var/lib/kubelet/config.yml" - "/etc/kubernetes/kubelet/kubelet-config.json" - "/etc/kubernetes/kubelet/config" - "/home/kubernetes/kubelet-config.yaml" - "/home/kubernetes/kubelet-config.yml" - "/etc/default/kubeletconfig.json" - "/etc/default/kubelet" - "/var/lib/kubelet/kubeconfig" - "/var/snap/kubelet/current/args" - "/var/snap/microk8s/current/args/kubelet" ## Due to the fact that the kubelet might be configured ## without a kubelet-config file, we use a work-around ## of pointing to the systemd service file (which can also ## hold kubelet configuration). ## Note: The following paths must match the one under 'svc' - "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" - "/etc/systemd/system/kubelet.service" - "/lib/systemd/system/kubelet.service" - "/etc/systemd/system/snap.kubelet.daemon.service" - "/etc/systemd/system/snap.microk8s.daemon-kubelet.service" - "/etc/kubernetes/kubelet.yaml" defaultconf: "/var/lib/kubelet/config.yaml" defaultsvc: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" defaultkubeconfig: "/etc/kubernetes/kubelet.conf" defaultcafile: "/etc/kubernetes/pki/ca.crt"

proxy: optional: true bins: - "kube-proxy" - "hyperkube proxy" - "hyperkube kube-proxy" - "proxy" - "openshift start network" confs: - /etc/kubernetes/proxy - /etc/kubernetes/addons/kube-proxy-daemonset.yaml - /etc/kubernetes/addons/kube-proxy-daemonset.yml - /var/snap/kube-proxy/current/args - /var/snap/microk8s/current/args/kube-proxy kubeconfig: - "/etc/kubernetes/kubelet-kubeconfig" - "/etc/kubernetes/kubelet-kubeconfig.conf" - "/etc/kubernetes/kubelet/config" - "/var/lib/kubelet/kubeconfig" - "/var/snap/microk8s/current/credentials/proxy.config" svc: - "/lib/systemd/system/kube-proxy.service" - "/etc/systemd/system/snap.microk8s.daemon-proxy.service" defaultconf: /etc/kubernetes/addons/kube-proxy-daemonset.yaml defaultkubeconfig: "/etc/kubernetes/proxy.conf"

etcd: components: - etcd

etcd: bins: - "etcd" confs: - /etc/kubernetes/manifests/etcd.yaml - /etc/kubernetes/manifests/etcd.yml - /etc/kubernetes/manifests/etcd.manifest - /etc/etcd/etcd.conf - /var/snap/etcd/common/etcd.conf.yml - /var/snap/etcd/common/etcd.conf.yaml - /var/snap/microk8s/current/args/etcd - /usr/lib/systemd/system/etcd.service defaultconf: /etc/kubernetes/manifests/etcd.yaml

controlplane: components: - apiserver

apiserver: bins: - "kube-apiserver" - "hyperkube apiserver" - "hyperkube kube-apiserver" - "apiserver"

policies: components: []

managedservices: components: []

version_mapping: "1.15": "cis-1.5" "1.16": "cis-1.6" "1.17": "cis-1.6" "1.18": "cis-1.6" "1.19": "cis-1.20" "1.20": "cis-1.20" "1.21": "cis-1.20" "1.22": "cis-1.23" "1.23": "cis-1.23" "eks-1.0.1": "eks-1.0.1" "eks-1.1.0": "eks-1.1.0" "gke-1.0": "gke-1.0" "gke-1.2.0": "gke-1.2.0" "ocp-3.10": "rh-0.7" "ocp-3.11": "rh-0.7" "ocp-4.0": "rh-1.0" "aks-1.0": "aks-1.0" "ack-1.0": "ack-1.0" "cis-1.6-k3s": "cis-1.6-k3s"

target_mapping: "cis-1.5": - "master" - "node" - "controlplane" - "etcd" - "policies" "cis-1.6": - "master" - "node" - "controlplane" - "etcd" - "policies" "cis-1.6-k3s": - "master" - "node" - "controlplane" - "etcd" - "policies" "cis-1.20": - "master" - "node" - "controlplane" - "etcd" - "policies" "cis-1.23": - "master" - "node" - "controlplane" - "etcd" - "policies" "gke-1.0": - "master" - "node" - "controlplane" - "etcd" - "policies" - "managedservices" "gke-1.2.0": - "master" - "node" - "controlplane" - "policies" - "managedservices" "eks-1.0.1": - "master" - "node" - "controlplane" - "policies" - "managedservices" "eks-1.1.0": - "master" - "node" - "controlplane" - "policies" - "managedservices" "rh-0.7": - "master" - "node" "aks-1.0": - "master" - "node" - "controlplane" - "policies" - "managedservices" "ack-1.0": - "master" - "node" - "controlplane" - "etcd" - "policies" - "managedservices" "rh-1.0": - "master" - "node" - "controlplane" - "policies" - "etcd" "eks-stig-kubernetes-v1r6": - "node" - "controlplane" - "policies" - "managedservices"`

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.] Pod Specification apiVersion: v1 kind: Pod metadata: name: kube-bench-pod namespace: kube-bench labels: name: kube-bench-pod spec: hostPID: true restartPolicy: Never containers: - name: kube-bench image: lnkdin.cr/temp/infosec/kube-bench:0.6.8 command: ["/bin/sh","-c"] args: ["kube-bench run -v 3 --targets node"] resources: requests: cpu: "2" memory: "2Gi" limits: cpu: "2" memory: "2Gi" volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: lib-systemd mountPath: /lib/systemd readOnly: true - name: srv-kubernetes mountPath: /srv/kubernetes readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true # /usr/local/mount-from-host/bin is mounted to access kubectl / kubelet, for auto-detecting the Kubernetes version. # You can omit this mount if you specify --version as part of the command. - name: usr-bin mountPath: /usr/local/mount-from-host/bin readOnly: true - name: config-volume mountPath: /opt/kube-bench/cfg readOnly: true volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: lib-systemd hostPath: path: "/lib/systemd" - name: srv-kubernetes hostPath: path: "/srv/kubernetes" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" - name: usr-bin hostPath: path: "/usr/bin" - name: config-volume configMap: name: my-config

[mozillazg]

@Algoss We can't use mount to overwrite config.yaml, because it will clean up all files and dirs under the /opt/kube-bench/cfg. The recommended way is to do it when building the docker image.

[Algoss]

@mozillazg I am not doing docker build. I am running this kube-bench pod in the k8s cluster. I have followed https://github.com/aquasecurity/kube-bench/issues/948#issuecomment-897489485 and tried the same solution. But I am not sure about the mountPath where the configMap has to be mounted. I just followed this https://github.com/aquasecurity/kube-bench/blob/main/docs/running.md#running-inside-a-container and used /opt/kube-bench/cfg/ but I feel it is wrong for my use case as I am running in the k8s cluster and not in the container.

I have asked the doubt here https://github.com/aquasecurity/kube-bench/issues/948#issuecomment-1320184136

Please provide your insights. Thanks