falcosecurity
version 0.33.1, chart 2.4.3 EKS: k8saudit-eks plugin does not load
Describe the bug
I have successfully deployed falco for syscall events on EKS and wanted to enable support for EKS audit events in the same deployment. However it does not work because k8saudit-eks and json plugins do not load. I followed the documentation https://falco.org/blog/k8saudit-eks-plugin/
How to reproduce it
Here are the configuration of the initial deployement:
falco:
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "sandbox-1"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [json, k8saudit-eks]
falcosidekick:
config:
customfields: "environment:sandbox,source:falco"
datadog:
apikey: ""
minimumpriority: warning
host: "https://api.datadoghq.com"
enabled: true
replicaCount: 2
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
webui:
enabled: true
replicaCount: 1
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
http_output:
enabled: true
url: "http://falcosidekick:2801/"
image:
tag: "0.33.1"
json_output: true
json_include_output_property: true
log_level: debug
resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
tty: true
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/FalcoAuditEKS
create: true
name: falco-audit-eks
customRules:
syscall_custom_rules.yaml: |-
There are no logs showing that the plugins are loading:
k logs -l app.kubernetes.io/name=falco -f
Thu Jan 5 13:40:53 2023: Falco version: 0.33.1 (x86_64)
Thu Jan 5 13:40:53 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Jan 5 13:40:53 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Jan 5 13:40:54 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Thu Jan 5 13:40:55 2023: Loading rules from file /etc/falco/rules.d/syscall_custom_rules.yaml
Thu Jan 5 13:40:56 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Jan 5 13:40:56 2023: Starting health webserver with threadiness 4, listening on port 8765
Thu Jan 5 13:40:56 2023: Enabled event sources: syscall
Thu Jan 5 13:40:56 2023: Opening capture with Kernel module
Thu Jan 5 13:40:56 2023: Falco version: 0.33.1 (x86_64)
Thu Jan 5 13:41:04 2023: Falco version: 0.33.1 (x86_64)
Thu Jan 5 13:41:04 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Jan 5 13:41:04 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Jan 5 13:41:05 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Thu Jan 5 13:40:56 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Jan 5 13:40:56 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Jan 5 13:40:57 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Thu Jan 5 13:40:58 2023: Loading rules from file /etc/falco/rules.d/syscall_custom_rules.yaml
Thu Jan 5 13:40:59 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Jan 5 13:40:59 2023: Starting health webserver with threadiness 4, listening on port 8765
Thu Jan 5 13:40:59 2023: Enabled event sources: syscall
Thu Jan 5 13:40:59 2023: Opening capture with Kernel module
Thu Jan 5 13:41:06 2023: Loading rules from file /etc/falco/rules.d/syscall_custom_rules.yaml
Thu Jan 5 13:41:07 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Jan 5 13:41:07 2023: Starting health webserver with threadiness 4, listening on port 8765
Thu Jan 5 13:41:07 2023: Enabled event sources: syscall
Thu Jan 5 13:41:07 2023: Opening capture with Kernel module
I tried to install on another cluster following the documentation https://github.com/falcosecurity/charts/tree/master/falco and have the same problem
driver:
enabled: false
collectors:
enabled: false
controller:
kind: deployment
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
region: "us-east-1"
profile: "default"
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "my-cluster"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
Expected behaviour
Plugins specified under load_plugins should load
Screenshots
Environment
- Falco version: 0.33.1
- System info: EKS 1.21/1.22
- Cloud provider or hardware configuration:
- OS: AmazonLinux2
- Kernel: 5.4.226-129.415.amzn2.x86_64
- Installation method: Helm
Additional context
Hi @rtalipov,
Sadly, the plugin is not yet available inside the image, as it has been released after the release of Falco. We're working on different approaches to have dynamic loadings.
For my personal usage, when I developed the plugin I used the official image as a From to create a new one, with just the .so added in /usr/share/falco/plugins.
Hi @Issif,
Glad to hear that you're working on loading the plugins.
I the meantime I've created a custom image with libk8saudit-eks.so under /usr/share/falco/plugins.
# cd /usr/share/falco/plugins
# ls
libcloudtrail.so libjson.so libk8saudit-eks.so libk8saudit.so
However the plugins still not loading with the config
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "sandbox-1"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [json, k8saudit-eks]
I've loaded custom rules and the pod is in CrashLoopBackOff with the error Error: Plugin requirement not satisfied, must load one of: k8saudit (>= 0.1.0), k8saudit-eks (>= 0.1.0)
Have you tried with the absolute path /usr/share/falco/plugins/k8saudit-eks ?
Take care, you can't use the k8saudit rules out of the box, you must update them with:
- required_engine_version: 15
- required_plugin_versions:
- name: k8saudit
version: 0.1.0
alternatives:
- name: k8saudit-eks
version: 0.1.0
as described here
I've tried the absolute path and it did not help. I guess I will be waiting for the dynamic loading feature in the future releases.
Looking at the Falco logs, the plugin is not being loaded at all, meaning that the plugin-related configuration does not reach Falco. What version of the our Helm Charts are you running?
cc @alacuku
@rtalipov, so at the moment we can't use the chart to monitor audit logs from clusters using Amazon EKS? I will also try it out with the custom Docker image.
I will hijack this issue and use this comment to document me trying to get it to work.
I download the plugin and built my own image. For now I am using the stable version of the plugin. I get it from here:
- https://download.falco.org/plugins/stable/k8saudit-eks-0.1.0-linux-x86_64.tar.gz
Building the image in the following temporary repo:
- https://github.com/trallnag/falco-with-plugins
I got the plugin to load!
Fri Jan 13 13:55:39 2023: Falco version: 0.33.1 (x86_64)
Fri Jan 13 13:55:39 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Jan 13 13:55:39 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Fri Jan 13 13:55:39 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Fri Jan 13 13:55:39 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Jan 13 13:55:39 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Fri Jan 13 13:55:40 2023: Starting health webserver with threadiness 4, listening on port 8765
Fri Jan 13 13:55:40 2023: Enabled event sources: k8s_audit
Fri Jan 13 13:55:40 2023: Opening capture with plugin 'k8saudit-eks'
Error: RequestError: send request failed
caused by: Post "https://logs.us-east-1.amazonaws.com/": x509: certificate signed by unknown authority
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
@rtalipov, maybe there is something wrong with your config?
Here is are my overrides:
image:
pullPolicy: Always
registry: ghcr.io/trallnag
repository: falco-with-plugins
tag: master
controller:
kind: deployment
driver:
enabled: false
collectors:
enabled: false
auditLog:
enabled: true
falco:
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
region: "us-east-1"
profile: "default"
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "c360-dev"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
Next I will add the certificates to the custom image. There is already a PR open for that by @Issif, but I don't want to wait.
Installing certs works.
Error: AccessDeniedException: User: arn:aws:sts::023551243526:assumed-role/c360-dev-eks-ng-apps/i-0efaa0f48d3cea5de is not authorized to perform: logs:FilterLogEvents on resource: arn:aws:logs:us-east-1:023551243526:log-group:/aws/eks/c360-dev/cluster:log-stream
Next I will configure permissions.
These seem to be enough:
{
"Statement": [
{
"Action": [
"logs:Describe*",
"logs:FilterLogEvents",
"logs:Get*",
"logs:List*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:logs:eu-central-1:023551243526:log-group:/aws/eks/CLUSTER_NAME/cluster:*"
],
"Sid": ""
}
],
"Version": "2012-10-17"
}
I have also overridden the included k8s audit rules because the images don't contain the latest versions.
Still not working. Now it is complaining about plugin requirements:
Fri Jan 13 21:50:51 2023: Falco version: 0.33.1-105+c1985a7 (x86_64)
Fri Jan 13 21:50:51 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Jan 13 21:50:51 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Fri Jan 13 21:50:51 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Fri Jan 13 21:50:51 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Jan 13 21:50:51 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Error: Plugin requirement not satisfied, must load one of: k8saudit (>= 0.1.0)
But I definitely have placed 0.1.0 version of plugin and the rules yaml looks right.
- required_engine_version: 15
- required_plugin_versions:
- name: k8saudit-eks
version: 0.1.0
- rule: Dummy rule
desc: >
Dummy rule
condition: >
ka.verb in (get,create,delete,update)
output: user=%ka.user.name verb=%ka.verb target=%ka.target.name target.namespace=%ka.target.namespace resource=%ka.target.resource
priority: WARNING
source: k8s_audit
tags: [k8s]
I finally figured it out. The chart is bringing it's own rules and it is overwriting the rules within the image.
I'm confused with this setup for eks plugin. Where the rules should be placed in the values.yaml or where to point to the k8s_audit_rules.yaml file? I already set up the plugin itself but when I try to deployed it I got this error:
falco restart count: 2
falco waiting reason: CrashLoopBackOff
falco termination reason: Error
falco-nt5jz/falco.log
Mon Mar 27 15:29:07 2023: Falco version: 0.34.1 (x86_64)
Mon Mar 27 15:29:07 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Mon Mar 27 15:29:07 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Mon Mar 27 15:29:07 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Error: You must specify at least one rules file/directory via -r or a rules_file entry in falco.yaml
configuration files resides in /rulesfiles dir and falco.yaml is empty:
# ls rulesfiles/
falco.yaml k8s_audit_rules.yaml
Here's my whole values.yaml I'm using for my demos:
tty: true
kubernetes: false
customRules:
override-k8saudit.yaml: |-
- list: allowed_k8s_users
append: true
items: [eks:cloud-controller-manager, eks:vpc-resource-controller, eks:az-poller]
- macro: live_endpoint
append: true
condition: or ka.uri="/readyz?exclude=kms-provider-0" or ka.uri="/livez?exclude=kms-provider-0"
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
region: "us-east-1"
# profile: "default"
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "falco-demo"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
json_output: true
json_include_output_property: true
json_include_tags_property: true
http_output:
enabled: true
url: "http://falco-falcosidekick.falco:2801"
driver:
enabled: false
collectors:
enabled: false
controller:
kind: deployment
deployment:
replicas: 1
falcoctl:
indexes:
- name: falcosecurity
url: https://falcosecurity.github.io/falcoctl/index.yaml
artifact:
install:
enabled: true
follow:
enabled: true
config:
artifact:
allowedTypes:
- plugin
- rulesfile
install:
resolveDeps: false
refs: [k8saudit-rules:0.5, k8saudit-eks:0.1, json:0]
follow:
refs: [k8saudit-rules:0.5]
I've a different daemonset for falco + falcosidekick + falcosidekick-ui, this is why I set also the json_* and http_output values.
I think your missing config is :
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
Here's my whole
values.yamlI'm using for my demos:tty: true kubernetes: false customRules: override-k8saudit.yaml: |- - list: allowed_k8s_users append: true items: [eks:cloud-controller-manager, eks:vpc-resource-controller, eks:az-poller] - macro: live_endpoint append: true condition: or ka.uri="/readyz?exclude=kms-provider-0" or ka.uri="/livez?exclude=kms-provider-0" falco: rules_file: - /etc/falco/k8s_audit_rules.yaml - /etc/falco/rules.d plugins: - name: k8saudit-eks library_path: libk8saudit-eks.so init_config: region: "us-east-1" # profile: "default" shift: 10 polling_interval: 10 use_async: false buffer_size: 500 open_params: "falco-demo" - name: json library_path: libjson.so init_config: "" load_plugins: [k8saudit-eks, json] json_output: true json_include_output_property: true json_include_tags_property: true http_output: enabled: true url: "http://falco-falcosidekick.falco:2801" driver: enabled: false collectors: enabled: false controller: kind: deployment deployment: replicas: 1 falcoctl: indexes: - name: falcosecurity url: https://falcosecurity.github.io/falcoctl/index.yaml artifact: install: enabled: true follow: enabled: true config: artifact: allowedTypes: - plugin - rulesfile install: resolveDeps: false refs: [k8saudit-rules:0.5, k8saudit-eks:0.1, json:0] follow: refs: [k8saudit-rules:0.5]I've a different daemonset for falco + falcosidekick + falcosidekick-ui, this is why I set also the
json_*andhttp_outputvalues.I think your missing config is :
falco: rules_file: - /etc/falco/k8s_audit_rules.yaml
@Issif, yes that was the case, now it's working, thanks a lot!
How to check if this plugin works properly? (that it's getting the logs from the cloudwatch eks logs)
@piteros83, try to do something that is checked by the default rules you have included in k8s_audit_rules.yaml. For example create a new namespace called "foobar". This should show up in the logs. There is a rule that checks this called "Create Disallowed Namespace".
@piteros83, try to do something that is checked by the default rules you have included in
k8s_audit_rules.yaml. For example create a new namespace called "foobar". This should show up in the logs. There is a rule that checks this called "Create Disallowed Namespace".
Creation of the new namespace on the cluster doesn't create any event in the falco. The only thing that I see right now in the sidekick-ui are the dropped syscalls events from the worker nodes. I found also this entry on one of the falco daemonset pod's:
Tue Mar 28 13:10:05 2023: Falco version: 0.34.1 (x86_64)
Tue Mar 28 13:10:05 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Tue Mar 28 13:10:05 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Tue Mar 28 13:10:05 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Tue Mar 28 13:10:05 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Tue Mar 28 13:10:05 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Tue Mar 28 13:10:05 2023: Starting health webserver with threadiness 4, listening on port 8765
Tue Mar 28 13:10:05 2023: Enabled event sources: k8s_audit, syscall
Tue Mar 28 13:10:05 2023: Opening capture with plugin 'k8saudit-eks'
Tue Mar 28 13:10:05 2023: Opening capture with Kernel module
**Wed Mar 29 00:43:43 2023: libcurl error: Couldn't resolve host name**
OK this could be the case:
2023-03-29 16:26:58 | caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-1.amazonaws.com/id/[...]
I had a typo in serviceAccount annotation (AWS account). Now I see all the events coming from k8s_adit source in sidekick-ui.
Do we have any reference how serviceAccount for IAM roles should be configured for Falco? I found only this tutorial analyze-aws-eks-audit-logs-with-falco. Maybe it's worth adding it to the Falco docs (or maybe I can do it by myself?)
Here's the minimal policy you need: https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit-eks#aws-iam-policy-permissions
Thx a lot again. Now I'm looking for tutorial how to write proper Exceptions. I went thru docs but it's unclear to me from where I should pick value for name key from example:
exceptions:
**- name: proc_writer**
fields: [proc.name, fd.directory]
comps: [=, =]
values:
- [my-custom-yum, /usr/bin]
- [my-custom-apt, /usr/local/bin]
You can use any value you want for "name" (alphanumeric chars and - work definitely). It is just used as an identifier for the respective exception.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community. /close
@poiana: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with
/reopen.Mark the issue as fresh with
/remove-lifecycle rotten.Provide feedback via https://github.com/falcosecurity/community. /close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.