falco
falco copied to clipboard
falcosecurity
Combination of default and customized macro not working
Describe the bug
I'm facing issues with combining custom macros with default macro. I created a new macro (combination_macro) combining my customized macro (known_exception) and pre-defined/default macro (write_etc_common). I created the below custom file and I executed vi /etc/falco/rules.d/random to create a random file. When I see the logs, I can see the output of known_exception and write_etc_common macro separately. But falco doesn't generate an output for the rule where I combine both the above macros into one combination_macro = (known_exception and write_etc_common). PFA output screenshot & my /etc/falco/rules.d/custom_rules.yaml.
How to reproduce it
- Install and start falco
- Create a new custom rule file,
/etc/falco/rules.d/custom_rules.yamlwith the below content
- macro: known_exception
condition: (user.name=root
and fd.name startswith /etc/falco/rules.d
and proc.name=vi)
- macro: combination_macro
condition: (known_exception
and write_etc_common)
- rule: Known Exception
desc: Known Exception an attempt to write to any file below /etc
condition: known_exception
output: "Known exception below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
- rule: Write below etc
desc: an attempt to write to any file below /etc
condition: write_etc_common
output: "Actual File below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
- rule: Combination
desc: Combination Test an attempt to write to any file below /etc
condition: combination_macro
output: "Combination below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
- Restart the service
- Open a new file in
viin/etc/falco/rules.ddirectory, write some data & save it. - View falco logs
Expected behaviour
According to expected behavior, all 3 rules should be executed: Known Exception, Write below etc, Combination because all the conditions are true but we get output only for Known Exception and Write below etc.
Screenshots
Environment
- Falco version: 0.29.1
- OS: Linux
- Installation method: apt-get install
/assign @fntlnz @shibumi
This is a quite simple operation, tried debugging for long hours but no progress. Please have a look & help.
@rewanthtammana: GitHub didn't allow me to assign the following users: shibumi.
Note that only falcosecurity members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide
In response to this:
/assign @fntlnz @shibumi
This is a quite simple operation, tried debugging for long hours but no progress. Please have a look & help.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
/assign @leogr @leodido
This is a quite simple operation, tried debugging for long hours but no progress. Please have a look & help.
mh I looked on this for a while now and I can't see the error as well. Are you sure it's not being logged? I think you forgot at least one line, because I see two errors being thrown for the swap file generated by vi and just one line for the actual file opened by vi. So there is at least one error line missing in your screenshot. Would be interesting to know if that gets thrown or not.
@shibumi You are right. I trimmed the output, the known_exception is being triggered twice while the default rule, write_etc_common is being executed only once. To make things easy, we can remove the proc.name=vi check in the above snippet. Use this updated snippet, restart falco and run touch yyy in /etc/falco/rules.d directory.
In output, known_exception = true, write_etc_common = true but known_exception and write_etc_common is not returning true.
/etc/falco/rules.d/custom_rules.yaml
- macro: known_exception
condition: (user.name=root
and fd.name startswith /etc/falco/rules.d)
- macro: combination_macro
condition: (known_exception
and write_etc_common)
- rule: Known Exception
desc: Known Exception an attempt to write to any file below /etc
condition: known_exception
output: "Known exception below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
- rule: Write below etc
desc: an attempt to write to any file below /etc
condition: write_etc_common
output: "Actual File below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
- rule: Combination
desc: Combination Test an attempt to write to any file below /etc
condition: combination_macro
output: "Combination below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
Trimmed logs required for this case
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875928818: Error Actual File below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875928818: Error Actual File below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875932381: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875933817: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875932381: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875933817: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Complete logs
controlplane $ journalctl | grep $(pidof falco)
Jul 18 04:41:55 controlplane falco[24559]: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Jul 18 04:41:55 controlplane falco[24559]: Sun Jul 18 04:41:55 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Jul 18 04:41:55 controlplane falco[24559]: Sun Jul 18 04:41:55 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Jul 18 04:41:55 controlplane falco[24559]: Sun Jul 18 04:41:55 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Jul 18 04:41:55 controlplane falco[24559]: Falco initialized with configuration file /etc/falco/falco.yaml
Jul 18 04:41:55 controlplane falco[24559]: Loading rules from file /etc/falco/falco_rules.yaml:
Jul 18 04:41:55 controlplane falco[24559]: Loading rules from file /etc/falco/falco_rules.local.yaml:
Jul 18 04:41:55 controlplane falco[24559]: Sun Jul 18 04:41:55 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Jul 18 04:41:55 controlplane falco[24559]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Jul 18 04:41:55 controlplane falco[24559]: Sun Jul 18 04:41:55 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Jul 18 04:41:56 controlplane falco[24559]: Loading rules from file /etc/falco/rules.d/custom_rules.yaml:
Jul 18 04:41:56 controlplane falco[24559]: Sun Jul 18 04:41:56 2021: Loading rules from file /etc/falco/rules.d/custom_rules.yaml:
Jul 18 04:41:56 controlplane falco[24559]: Rule Known Exception: warning (no-evttype):
Jul 18 04:41:56 controlplane falco[24559]: known_exception
Jul 18 04:41:56 controlplane falco[24559]: did not contain any evt.type restriction, meaning it will run for all event types.
Jul 18 04:41:56 controlplane falco[24559]: This has a significant performance penalty. Consider adding an evt.type restriction if possible.
Jul 18 04:41:56 controlplane falco[24559]: Starting internal webserver, listening on port 8765
Jul 18 04:41:56 controlplane falco[24559]: Sun Jul 18 04:41:56 2021: Starting internal webserver, listening on port 8765
Jul 18 04:41:56 controlplane falco[24559]: 04:41:56.429410000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:ce49d3a4f31d k8s_weave-npc_weave-net-x5b2j_kube-system_46002bae-e777-11eb-93ed-0242ac110018_0 (id=ce49d3a4f31d) image=weaveworks/weave-npc:2.6.0)
Jul 18 04:41:56 controlplane falco[24559]: 04:41:56.432954000: Notice Privileged container started (user=root user_loginuid=0 command=container:9043d686d32d k8s_weave_weave-net-x5b2j_kube-system_46002bae-e777-11eb-93ed-0242ac110018_1 (id=9043d686d32d) image=weaveworks/weave-kube:2.6.0)
Jul 18 04:41:56 controlplane falco[24559]: 04:41:56.429410000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:ce49d3a4f31d k8s_weave-npc_weave-net-x5b2j_kube-system_46002bae-e777-11eb-93ed-0242ac110018_0 (id=ce49d3a4f31d) image=weaveworks/weave-npc:2.6.0)
Jul 18 04:41:56 controlplane falco[24559]: 04:41:56.432954000: Notice Privileged container started (user=root user_loginuid=0 command=container:9043d686d32d k8s_weave_weave-net-x5b2j_kube-system_46002bae-e777-11eb-93ed-0242ac110018_1 (id=9043d686d32d) image=weaveworks/weave-kube:2.6.0)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875928818: Error Actual File below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875928818: Error Actual File below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875932381: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875933817: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875932381: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
Jul 18 04:41:57 controlplane falco[24559]: 04:41:57.875933817: Error Known exception below /etc opened for writing (user=root user_loginuid=0 command=touch yyy parent=bash pcmdline=bash file=/etc/falco/rules.d/yyy program=touch gparent=sshd ggparent=sshd gggparent=systemd container_id=host image=<NA>)
controlplane $
I am new to falco, but is it possible that the first rule matches and that's it? What I am trying to say is that I think it might make sense that the rule gets only triggered alone, because both conditions of the multi-macro has been already triggered. So why triggering that multi macro at all (both parts have been already triggered).
Can you validate this?
- macro: known_exception
condition: (user.name=root
and fd.name startswith /etc/falco/rules.d)
- macro: combination_macro
condition: (known_exception
and write_etc_common)
- rule: Combination
desc: Combination Test an attempt to write to any file below /etc
condition: combination_macro
output: "Combination below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
priority: ERROR
tags: [filesystem, mitre_persistence]
Does the above rule work?
I tried this already, it's not working. To find the root cause, I added granular rules for each of the above conditions to understand where the mistake is happening but still no progress.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Sorry, I completely lost track of this. Anyway, after reading the description again, it does not seem like a bug to me. :thinking:
rule: Combination will never be triggered because it is shadowed by the other two rules. When an event arrives, Falco evaluates it against each rule and stops on the first match.
/cc @jasondellaluce
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten
I support Leo's point. This is how Falco works by design, it triggers the first rule that matches the given event. This happens so that the rule engine can be as fast as possible. Notably, this often introduces a "shadowing" effect where a rule condition will prevent another rule from ever being triggered depending on the order they are defined, just because the rule engine tried to match all the rules in that order.
My proposal is to close this issue, as this is a side-effect we have by design. If we want to achieve something different, I'd suggest creating a feature-request issue or a even a proposal, and eventually link this issue there. I'll wait a little while to see what everyone think before closing.
Perhaps we should improve the documentation, just to avoid headaches :)
Wdty?
@leogr Improving documentation will definitely save lots of time for other folks. I spent many hours trying to debug it.
@jasondellaluce But when it comes to Falco, there will be hundreds/thousands of rules. Having the FCFS model might be tricky for people to manage. What if they aren't aware of the presence of the above rule? I think this should be raised as a feature request to resolve all kinds of confusion.
@rewanthtammana If that's the case, would you consider closing this issue and opening a feature request one?
As for the reasons behind this Falco design choice, you have to consider that Falco consumes a very large number of events per second. Among the CPU-time-expensive tasks Falco does, ruleset evaluation for each event is definitely the most costly we have. By stopping the ruleset evaluation at the first rule triggered by an event, we save plenty of precious CPU-time that allows Falco to sustain a high throughput of events. If you look at the default ruleset, you'll notice that most rules are designed to "fail fast" for this reason specifically, and I also bet that in many cases folks hitting an high drop rate didn't tune their custom ruleset in the right way. The faster Falco is able to consume events, the less CPU is used and events are dropped. Maintaining an event drop rate as close as possible to 0, and a low CPU usage overhead, are both top priorities for Falco in order to sustain its security guarantees.
As such, changing the rule triggering policy to keep looking for rule matches even after the first one would be very expensive and would have an high impact on Falco performance and security guarantees. This being said, I still think the issue you are reporting here is a matter of writing "correct rulesets" given the Falco design principles. I agree with you, we could be more explicit in the community about how rulesets should be written. Leaving here few ideas:
- Better documentation on the website (OFC)
- Make Falco look for potential rule shadowing like your case, and print warnings when it notices some potential mistake
- Provide IDE-like support (event online or with something like a VS-code plugin) for Falco rulesets to help folks get up to speed with ruleset development
@jasondellaluce I agree. The event count is quite high & I have observed the packet drops as well.
I also bet that in many cases folks hitting an high drop rate didn't tune their custom ruleset in the right way.
Is there any guide or workflow on how to optimize the custom ruleset(s)?
I like your ideas.
Make Falco look for potential rule shadowing like your case, and print warnings when it notices some potential mistake
Finding a potential mistake/duplicate might be challenging. Just a suggestion, I'm not sure of the technical complexity in implementation. As a temporary fix, we can have the logs printed with the rule number. This allows the users to know that Falco stopped matching rules at maybe rule number 400 & it won't reach the user's custom rule which is at rule number 1200. I believe this will make debugging much easier.
Provide IDE-like support (event online or with something like a VS-code plugin) for Falco rulesets to help folks get up to speed with ruleset development
A VS-code plugin would be amazing. I would love to collaborate on its development anytime.
An additional idea:
To provide flexibility to the end-users, we can add an option in Falco's configuration to ignore FCFS policy. This will allow the end-user to perform all the tests on rule sets without thinking twice about efficiency. I don't see such an option exists at the moment.
We can have the default Falco configuration as fail fast but let the user have the flexibility to tweak their configuration & choose how they want to work. Let me know your thoughts.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community. /close
@poiana: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with
/reopen.Mark the issue as fresh with
/remove-lifecycle rotten.Provide feedback via https://github.com/falcosecurity/community. /close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Now addressed by https://github.com/falcosecurity/falco/pull/2705 from @loresuso.
Awesome. I didn't expect it to become a feature as it was left unanswered. This looks great & excited to try it @jasondellaluce