Falco with plugin k8saudit-eks

[pealtrufo]

I am a little bit confused when following documentation to deploy Falco on an EKS cluster with the kernel module and also the k8saudit-eks plugin.

I am deploying it using helm, and using the yaml file in chart repo below as a basis, where I make the changes required to use the k8saudit-eks plugin instead https://github.com/falcosecurity/charts/blob/master/falco/values-syscall-k8saudit.yaml https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit-eks

The resulting yaml I am using is this:

# Enable the driver, and choose between the kernel module or the ebpf probe.
# Default value: kernel module.
driver:
  enabled: true
  kind: module

# Enable the collectors used to enrich the events with metadata.
# Check the values.yaml file for fine-grained options.
collectors:
  enabled: true

# We set the controller to daemonset since we have the syscalls source enabled.
# It will ensure that every node on our cluster will be monitored by Falco.
# Please note that the api-server will use the "k8saudit-webhook" service to send
# audit logs to the falco instances. That means that when we have multiple instances of Falco
# we can not predict to which instance the audit logs will be sent. When testing please check all
# the Falco instance to make sure that at least one of them have received the audit logs.
controller:
  kind: daemonset

tty: true

falcoctl:
  artifact:
    install:
      # -- Enable the init container. We do not recommend installing plugins for security reasons since they are executable objects.
      # We install only "rulesfiles".
      enabled: true
    follow:
      # -- Enable the sidecar container. We do not support it yet for plugins. It is used only for rules feed such as k8saudit-rules rules.
      enabled: true
  config:
    artifact:
      install:
        # -- Do not resolve the depenencies for artifacts. By default is true, but for our use case we disable it.
        resolveDeps: false
        # -- List of artifacts to be installed by the falcoctl init container.
        # We do not recommend installing (or following) plugins for security reasons since they are executable objects.
        refs: [falco-rules:0, k8saudit-rules:0.5]
      follow:
        # -- List of artifacts to be followed by the falcoctl sidecar container.
        # We do not recommend installing (or following) plugins for security reasons since they are executable objects.
        refs: [falco-rules:0, k8saudit-rules:0.5]

services:
  - name: k8saudit-webhook
    type: NodePort
    ports:
      - port: 9765 # See plugin open_params
        nodePort: 30007
        protocol: TCP

falco:
  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  jsonOutput: true
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        region: "eu-west-2"
        profile: "default"
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "cluster-name"
    - name: json
      library_path: libjson.so
      init_config: ""

  load_plugins: [k8saudit-eks, json]

Now when deploying this config, Falco pods can't start and these are the logs I can see:

kubectl logs -n falco falco-2m445
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Thu Mar 30 15:06:26 2023: Falco version: 0.34.1 (x86_64)
Thu Mar 30 15:06:26 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Mar 30 15:06:26 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Thu Mar 30 15:06:26 2023: Runtime error: cannot load plugin /usr/share/falco/plugins/libk8saudit-eks.so: can't load plugin dynamic library: /usr/share/falco/plugins/libk8saudit-eks.so: cannot open shared object file: No such file or directory. Exiting.

I can see this warning in the output of the helm install command:

WARNING:
It seems you are loading the following plugins [k8saudit-eks json], please make sure to install them by adding the correct reference to falcoctl.config.artifact.install.refs: [falco-rules:0 k8saudit-rules:0.5]

But when looking to the sample yaml, there's this comment: We do not recommend installing (or following) plugins for security reasons since they are executable objects

  config:
    artifact:
      install:
        # -- Do not resolve the depenencies for artifacts. By default is true, but for our use case we disable it.
        resolveDeps: false
        # -- List of artifacts to be installed by the falcoctl init container.
        # We do not recommend installing (or following) plugins for security reasons since they are executable objects.
        refs: [falco-rules:0, k8saudit-rules:0.5]

Which seems to imply that you don't recommend installing plugins this way.

If I amend the yaml file and configure the falcoctl section this way, pods are started ok:

falcoctl:
  artifact:
    install:
      # -- Enable the init container. We do not recommend installing plugins for security reasons since they are executable objects.
      # We install only "rulesfiles".
      enabled: true
    follow:
      # -- Enable the sidecar container. We do not support it yet for plugins. It is used only for rules feed such as k8saudit-rules rules.
      enabled: true
  config:
    artifact:
      allowedTypes:
        - rulesfile
        - plugin
      install:
        # -- Do not resolve the depenencies for artifacts. By default is true, but for our use case we disable it.
        resolveDeps: false
        # -- List of artifacts to be installed by the falcoctl init container.
        # We do not recommend installing (or following) plugins for security reasons since they are executable objects.
        refs: [falco-rules:0, k8saudit-rules:0.5, k8saudit-eks:0.1, json:0.6]
        # refs: [falco-rules:0, k8saudit-rules:0.5]
      follow:
        # -- List of artifacts to be followed by the falcoctl sidecar container.
        # We do not recommend installing (or following) plugins for security reasons since they are executable objects.
        refs: [falco-rules:0, k8saudit-rules:0.5]

Note I added plugins to be installed in refs also had to add value plugin as allowedType in config for it to work. Otherwise, pods wouldn't start.

Is this ok? I ended up with this config based on googling and testing different configs I found. I don't think it is clear in the docs so I suppose I am looking for validation :) I am happy to contribute with any required changes to the docs if you think that's required.

This is what I can see in the logs just after deploy. So everything seems to be ok. I can trigger rules too and can see the warnings in pods output:

kubectl logs -n falco falco-k4vfp
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Thu Mar 30 15:21:44 2023: Falco version: 0.34.1 (x86_64)
Thu Mar 30 15:21:44 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Mar 30 15:21:44 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Thu Mar 30 15:21:44 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Thu Mar 30 15:21:44 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Mar 30 15:21:44 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Thu Mar 30 15:21:44 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Mar 30 15:21:44 2023: Starting health webserver with threadiness 2, listening on port 8765
Thu Mar 30 15:21:44 2023: Enabled event sources: k8s_audit, syscall
Thu Mar 30 15:21:44 2023: Opening capture with plugin 'k8saudit-eks'
Thu Mar 30 15:21:44 2023: Opening capture with Kernel module

Note: I understand the k8s service is not required when using k8saudit-eks plugin, right?

Environment

• Falco version: 0.34.1 (x86_64)
• Cloud provider or hardware configuration: AWS EKS
• Installation method: Helm

Thanks!

[Issif]

Hi,

I agree the documentation is not clear enough, I'll try to update the README asap.

You should not collect syscalls + eks audit logs with same the Falco pods, in fact, you should run only ONE pod with the k8saudit-eks plugin. The reason is simple, to work, the plugin pulls Cloudwatch Logs. By having several pods, it means several Falco will collect same logs and the alerts will be duplicated.

Here's a snippet of the values I use to deploy Falco + k8saudit-eks plugin:

tty: true
kubernetes: false

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        region: "us-east-1"
        # profile: "default"
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "falco-demo"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]
  json_output: true
  json_include_output_property: true
  json_include_tags_property: true
  http_output:
    enabled: true
    url: "http://falco-falcosidekick.falco:2801"

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0.5, k8saudit-eks:0.1, json:0]
      follow:
        refs: [k8saudit-rules:0.5]

It deploys only 1 pod:

controller:
  kind: deployment
  deployment:
    replicas: 1

And the events are sent to the falcosidekick service of my classic daemonset installation of Falco for syscalls.

  json_output: true
  json_include_output_property: true
  json_include_tags_property: true
  http_output:
    enabled: true
    url: "http://falco-falcosidekick.falco:2801"

[pealtrufo]

Thanks @Issif that makes sense 👍

Then, two different Falco installations would be recommended if you want to collect syscalls and EKS k8s audit events?

[Issif]

With EKS clusters yes. One daemonset to collect syscalls, and a 1 replica deployment for Falco + k8saudit-eks plugin to collect audit logs of EKS. My snippet shows how to disable the syscalls collection

[pealtrufo]

@Issif I would suggest adding to the documentation that you can configure IRSA for Falco service account in EKS cluster so that you don't have to attach the IAM policy described in the documentation to the cluster nodes. The error message you get when access to cloudwatch is not configured for Falco seems to imply that the nodegroup don't have that access so perhaps that could be confusing.

I have just tested it and works ok so far 👍

[Issif]

If you use an Instance Profile or IRSA, in both cases, you need to attach the IAM Policy to a Role.

[pealtrufo]

Yes, I understand. I meant that the error message could be confusing and make some people attach the role to the nodes rather than the service account, which is not ideal.

[Issif]

I agree, I just thought people who use EKS + Falco + k8saudit-eks are familiar enough with AWS Best Practices :sweat_smile: If you have time, you can also propose the PR to update the README by yourself, thanks.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[Issif]

/close

[poiana]

@Issif: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.