falco
falco copied to clipboard
falcosecurity
Falco keeps restarting when run with ebpf driver in least privilege mode
Describe the bug When I run falco as a demonset in least priviledge mode using the edpf driver on a red hat enterprise linux 8.9, I see multiple falco process restarts in the logs on a kubernetes environment. When I run falco with the edpf driver in privilege mode, it works fine, however running in least privilege results in multiple restarts.
How to reproduce it Add the following capabilities to the security context in the deployment.yaml file
securityContext:
capabilities:
add:
- SYS_ADMIN
- SYS_RESOURCE
- SYS-PTRACE
Logs shows
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco-exporter
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falcosidekick
SUPERVISOR: 2024/02/20 16:43:12 Starting reaper
SUPERVISOR: 2024/02/20 16:43:12 Running `/usr/bin/run.sh`
2024-02-20 16:43:12 INFO Running falcoctl driver config
├ name: falco
├ version: 7.0.0+driver
├ type: ebpf
├ host-root: /host
└ repos: https://download.falco.org/driver
2024-02-20 16:43:12 WARN Avoid updating Falco configuration
├ config: /etc/falco/falco.yaml
└ reason: engine.kind is not driver driven:
2024-02-20 16:43:12 INFO Running falcoctl driver install
├ driver version: 7.0.0+driver
├ driver type: ebpf
├ driver name: falco
├ compile: true
├ download: true
├ arch: x86_64
├ kernel release: 4.18.0-513.11.1.el8_9.x86_64
└ kernel version: #1 SMP Thu Dec 7 03:06:13 EST 2023
2024-02-20 16:43:12 INFO Found distro target: rhel
2024-02-20 16:43:12 INFO Removing eBPF probe symlink
└ path: /root/.falco/falco-bpf.o
2024-02-20 16:43:12 INFO Trying to download a driver.
└ url: https://download.falco.org/driver/7.0.0%2Bdriver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:13 WARN Non-200 response from url. code: 404
2024-02-20 16:43:13 WARN unable to find a prebuilt driver
2024-02-20 16:43:13 INFO Mounting debugfs for bpf driver.
2024-02-20 16:43:30 INFO Copying built driver to its destination.
├ src: /usr/src/falco-7.0.0+driver/bpf/probe.o
└ dst: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO Driver built.
└ path: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO Symlinking eBPF probe
├ src: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
└ dest: /root/.falco/falco-bpf.o
2024-02-20 16:43:30 INFO eBPF probe symlinked
SUPERVISOR: 2024/02/20 16:43:30 Completed `/usr/bin/run.sh`
SUPERVISOR: 2024/02/20 16:43:30 Starting falco
SUPERVISOR: 2024/02/20 16:43:30 falco started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falco-exporter
SUPERVISOR: 2024/02/20 16:43:30 falco-exporter started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falcosidekick
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:31 falcosidekick started successfully
SUPERVISOR: 2024/02/20 16:43:41 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:51 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:01 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:11 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:21 falco process has stopped, restarting.
Expected behaviour
I will like falco to run in least priviledge mode on a red hat enterprise linux 8.9 machine without multiple restarts.
Environment
- Falco version: 0.38
- OS: Rhel
- Installation method: Deployment.yaml in a kubernetes environment
@Nnoromuche There are some possible reasons for this issue, in my case when I was running Falco in the least privileged mode it didn't have the necessary permissions to load the eBPF driver. If we use Pod Security policies in the Kubernetes cluster, they might restrict the capabilities of the Falco pods, preventing them from loading the eBPF driver.
Here are the key permissions required to run Falco with eBPF driver:
- SYS_ADMIN_Capability
- Host PID Namespace
- Host Networking
- Volume Mounts
You can check whether these necessary permissions are given or not in Kubernetes least privileged mode.
@rashim27us thanks, I am running falco with the below deployment.yaml file and the SYS_ADMIN_Capability is being set.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: test-falco
namespace: test
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 10%
selector:
matchLabels:
app: test-falco
template:
metadata:
name: test-falco
namespace: test
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "12076"
prometheus.istio.io/merge-metrics: "false"
sidecar.istio.io/inject: "false"
labels:
app: test-falco
addonmanager.kubernetes.io/mode: Reconcile
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
tolerations:
- operator: Exists
volumes:
- name: dshm
emptyDir:
medium: Memory
- name: containerd-socket
hostPath:
path: /run/containerd/containerd.sock
- name: proc-fs
hostPath:
path: /proc
- name: boot-fs
hostPath:
path: /boot
- name: lib-modules
hostPath:
path: /lib/modules
- name: usr-fs
hostPath:
path: /usr
- name: etc-fs
hostPath:
path: /etc
- name: root-falco-fs
emptyDir: {}
containers:
- name: test-falco
image: (( concat "{{ FALCO_IMAGE_REPO_HOSTNAME}}/test/test-falco:" metadata.annotations.version ))
env:
- name: HOST_ROOT
value: "/host"
securityContext:
capabilities:
drop:
- all
add:
- SYS_ADMIN
- SYS_RESOURCE
- SYS_PTRACE
livenessProbe:
httpGet:
path: /liveness
port: 12077
periodSeconds: 60
failureThreshold: 10
readinessProbe:
httpGet:
path: /readiness
port: 12077
periodSeconds: 60
failureThreshold: 10
volumeMounts:
- mountPath: /host/run/containerd/containerd.sock
name: containerd-socket
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
readOnly: false
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /dev/shm
name: dshm
- mountPath: /host/etc
name: etc-fs
readOnly: true
- mountPath: /root/.falco
name: root-falco-fs
resources:
requests:
cpu: 25m
memory: 230Mi
limits:
memory: 400Mi
The edpf driver is installed using falcoctl commands
/usr/bin/falcoctl driver config --type ebpf
/usr/bin/falcoctl driver install --compile --download
Wondering is there a missing configuration in the above deployment.yaml file?
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
