tracee icon indicating copy to clipboard operation
tracee copied to clipboard

Published 20 hours ago • verified publisher icon aquasecurity

When enabling "exec-env" option, too many false positives happen

Open rafaeldtinoco opened this issue 2 years ago • 7 comments

Description

When running tracee in a k8s environment, if I change the default configmap to have "exec-env" option enabled, I started getting too many "ld_preload" signatures triggered for many processes:

$ microk8s.kubectl describe cm -n default tracee-config 
...
Data
====
config.yaml:
----
no-containers: false
cache:
    type: mem
    size: 512
perf-buffer-size: 1024
healthz: true
metrics: true
pprof: false
pyroscope: false
listen-addr: :3366
log:
    level: info
output:
    json:
      files:
        - stdout
    options:
        parse-arguments: true
        stack-addresses: false
        exec-env: true
        relative-time: true
        exec-hash: false
        sort-events: true
...

The false positives:

{"timestamp":310041368905,"threadStartTime":310040881916,"processorId":3,"processId":30736,"cgroupId":2751,"threadId":30736,"parentProcessId":595,"hostProcessId":30736,"hostThreadId":30736,"hostParentProcessId":595,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"snapctl","executable":{"path":""},"hostName":"vm02","containerId":"","container":{},"kubernetes":{},"eventId":"6006","eventName":"ld_preload","matchedPolicies":["default-policy"],"argsNum":2,"returnValue":0,"syscall":"","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":0,"processEntityId":0,"parentEntityId":0,"args":[{"name":"LD_LIBRARY_PATH","type":"const char *","value":"LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu"},{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"cmdpath","type":"const char*","value":"/usr/bin/snapctl"},{"name":"pathname","type":"const char*","value":"/usr/lib/snapd/snapctl"},{"name":"dev","type":"dev_t","value":8388609},{"name":"inode","type":"unsigned long","value":16793},{"name":"ctime","type":"unsigned long","value":1697595129754118964},{"name":"inode_mode","type":"umode_t","value":33261},{"name":"interpreter_pathname","type":"const char*","value":null},{"name":"interpreter_dev","type":"dev_t","value":null},{"name":"interpreter_inode","type":"unsigned long","value":null},{"name":"interpreter_ctime","type":"unsigned long","value":null},{"name":"argv","type":"const char**","value":["snapctl","services","microk8s.daemon-kubelite"]},{"name":"interp","type":"const char*","value":"/usr/bin/snapctl"},{"name":"stdin_type","type":"string","value":"S_IFCHR"},{"name":"stdin_path","type":"char*","value":"/dev/null"},{"name":"invoked_from_kernel","type":"int","value":0},{"name":"env","type":"const char**","value":["SNAP_REVISION=6070","SNAP_REAL_HOME=/root","SNAP_USER_COMMON=/root/snap/microk8s/common","SNAP_INSTANCE_KEY=","SNAP_EUID=0","PWD=/var/snap/microk8s/6070","SYSTEMD_EXEC_PID=595","SNAP_CONTEXT=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","LANG=C.UTF-8","SNAP_ARCH=amd64","SNAP_INSTANCE_NAME=microk8s","SNAP_USER_DATA=/root/snap/microk8s/6070","INVOCATION_ID=0549ddae482040868d7df7a41ddf1ce2","SNAP_REEXEC=","SNAP_UID=0","PYTHONPATH=/snap/microk8s/6070/usr/lib/python3.8:/snap/microk8s/6070/lib/python3.8/site-packages:/snap/microk8s/6070/usr/lib/python3/dist-packages:","SNAP=/snap/microk8s/6070","SNAP_COMMON=/var/snap/microk8s/common","SNAP_VERSION=v1.27.6","SHLVL=0","SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void","SNAP_COOKIE=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","SNAP_DATA=/var/snap/microk8s/6070","LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu","LC_ALL=C.UTF-8","SNAP_NAME=microk8s","JOURNAL_STREAM=8:15068","PATH=/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/sbin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin","_=/usr/bin/snapctl"]}],"id":713,"name":"sched_process_exec","returnValue":0}}],"metadata":{"Version":"1","Description":"LD_PRELOAD usage was detected. LD_PRELOAD lets you load your library before any other library, allowing you to hook functions in a process. Adversaries may use this technique to change your applications' behavior or load their own programs.","Tags":null,"Properties":{"Category":"persistence","Kubernetes_Technique":"","Severity":2,"Technique":"Hijack Execution Flow","external_id":"T1574","id":"attack-pattern--aedfca76-3b30-4866-b2aa-0f1d7fd1e4b6","signatureID":"TRC-107","signatureName":"LD_PRELOAD code injection detected"}}}
{"timestamp":310041420783,"threadStartTime":310040974073,"processorId":0,"processId":30737,"cgroupId":2751,"threadId":30737,"parentProcessId":595,"hostProcessId":30737,"hostThreadId":30737,"hostParentProcessId":595,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"grep","executable":{"path":""},"hostName":"vm02","containerId":"","container":{},"kubernetes":{},"eventId":"6006","eventName":"ld_preload","matchedPolicies":["default-policy"],"argsNum":2,"returnValue":0,"syscall":"","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":0,"processEntityId":0,"parentEntityId":0,"args":[{"name":"LD_LIBRARY_PATH","type":"const char *","value":"LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu"},{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"cmdpath","type":"const char*","value":"/snap/microk8s/6070/bin/grep"},{"name":"pathname","type":"const char*","value":"/snap/microk8s/6070/bin/grep"},{"name":"dev","type":"dev_t","value":7340037},{"name":"inode","type":"unsigned long","value":414},{"name":"ctime","type":"unsigned long","value":1696933677000000000},{"name":"inode_mode","type":"umode_t","value":33261},{"name":"interpreter_pathname","type":"const char*","value":"/snap/core20/2015/usr/lib/x86_64-linux-gnu/ld-2.31.so"},{"name":"interpreter_dev","type":"dev_t","value":7340033},{"name":"interpreter_inode","type":"unsigned long","value":6046},{"name":"interpreter_ctime","type":"unsigned long","value":1649294681000000000},{"name":"argv","type":"const char**","value":["grep","active"]},{"name":"interp","type":"const char*","value":"/snap/microk8s/6070/bin/grep"},{"name":"stdin_type","type":"string","value":"S_IFIFO"},{"name":"stdin_path","type":"char*","value":""},{"name":"invoked_from_kernel","type":"int","value":0},{"name":"env","type":"const char**","value":["SNAP_REVISION=6070","SNAP_REAL_HOME=/root","SNAP_USER_COMMON=/root/snap/microk8s/common","SNAP_INSTANCE_KEY=","SNAP_EUID=0","PWD=/var/snap/microk8s/6070","SYSTEMD_EXEC_PID=595","SNAP_CONTEXT=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","LANG=C.UTF-8","SNAP_ARCH=amd64","SNAP_INSTANCE_NAME=microk8s","SNAP_USER_DATA=/root/snap/microk8s/6070","INVOCATION_ID=0549ddae482040868d7df7a41ddf1ce2","SNAP_REEXEC=","SNAP_UID=0","PYTHONPATH=/snap/microk8s/6070/usr/lib/python3.8:/snap/microk8s/6070/lib/python3.8/site-packages:/snap/microk8s/6070/usr/lib/python3/dist-packages:","SNAP=/snap/microk8s/6070","SNAP_COMMON=/var/snap/microk8s/common","SNAP_VERSION=v1.27.6","SHLVL=0","SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void","SNAP_COOKIE=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","SNAP_DATA=/var/snap/microk8s/6070","LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu","LC_ALL=C.UTF-8","SNAP_NAME=microk8s","JOURNAL_STREAM=8:15068","PATH=/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/sbin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin","_=/snap/microk8s/6070/bin/grep"]}],"id":713,"name":"sched_process_exec","returnValue":0}}],"metadata":{"Version":"1","Description":"LD_PRELOAD usage was detected. LD_PRELOAD lets you load your library before any other library, allowing you to hook functions in a process. Adversaries may use this technique to change your applications' behavior or load their own programs.","Tags":null,"Properties":{"Category":"persistence","Kubernetes_Technique":"","Severity":2,"Technique":"Hijack Execution Flow","external_id":"T1574","id":"attack-pattern--aedfca76-3b30-4866-b2aa-0f1d7fd1e4b6","signatureID":"TRC-107","signatureName":"LD_PRELOAD code injection detected"}}}
{"timestamp":310049980862,"threadStartTime":310049704987,"processorId":3,"processId":30743,"cgroupId":2751,"threadId":30743,"parentProcessId":595,"hostProcessId":30743,"hostThreadId":30743,"hostParentProcessId":595,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"sleep","executable":{"path":""},"hostName":"vm02","containerId":"","container":{},"kubernetes":{},"eventId":"6006","eventName":"ld_preload","matchedPolicies":["default-policy"],"argsNum":2,"returnValue":0,"syscall":"","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":0,"processEntityId":0,"parentEntityId":0,"args":[{"name":"LD_LIBRARY_PATH","type":"const char *","value":"LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu"},{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"cmdpath","type":"const char*","value":"/snap/microk8s/6070/bin/sleep"},{"name":"pathname","type":"const char*","value":"/snap/microk8s/6070/bin/sleep"},{"name":"dev","type":"dev_t","value":7340037},{"name":"inode","type":"unsigned long","value":445},{"name":"ctime","type":"unsigned long","value":1696933680000000000},{"name":"inode_mode","type":"umode_t","value":33261},{"name":"interpreter_pathname","type":"const char*","value":"/snap/core20/2015/usr/lib/x86_64-linux-gnu/ld-2.31.so"},{"name":"interpreter_dev","type":"dev_t","value":7340033},{"name":"interpreter_inode","type":"unsigned long","value":6046},{"name":"interpreter_ctime","type":"unsigned long","value":1649294681000000000},{"name":"argv","type":"const char**","value":["sleep","5"]},{"name":"interp","type":"const char*","value":"/snap/microk8s/6070/bin/sleep"},{"name":"stdin_type","type":"string","value":"S_IFCHR"},{"name":"stdin_path","type":"char*","value":"/dev/null"},{"name":"invoked_from_kernel","type":"int","value":0},{"name":"env","type":"const char**","value":["SNAP_REVISION=6070","SNAP_REAL_HOME=/root","SNAP_USER_COMMON=/root/snap/microk8s/common","SNAP_INSTANCE_KEY=","SNAP_EUID=0","PWD=/var/snap/microk8s/6070","SYSTEMD_EXEC_PID=595","SNAP_CONTEXT=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","LANG=C.UTF-8","SNAP_ARCH=amd64","SNAP_INSTANCE_NAME=microk8s","SNAP_USER_DATA=/root/snap/microk8s/6070","INVOCATION_ID=0549ddae482040868d7df7a41ddf1ce2","SNAP_REEXEC=","SNAP_UID=0","PYTHONPATH=/snap/microk8s/6070/usr/lib/python3.8:/snap/microk8s/6070/lib/python3.8/site-packages:/snap/microk8s/6070/usr/lib/python3/dist-packages:","SNAP=/snap/microk8s/6070","SNAP_COMMON=/var/snap/microk8s/common","SNAP_VERSION=v1.27.6","SHLVL=1","SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void","SNAP_COOKIE=lGjw0Qm7x4cgAEd0QIOdClvVq4s-3mTB01eiuJmO5YMX9UAtzf1j","SNAP_DATA=/var/snap/microk8s/6070","LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6070/lib:/snap/microk8s/6070/usr/lib:/snap/microk8s/6070/lib/x86_64-linux-gnu:/snap/microk8s/6070/usr/lib/x86_64-linux-gnu","LC_ALL=C.UTF-8","SNAP_NAME=microk8s","JOURNAL_STREAM=8:15068","PATH=/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/sbin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/bin:/snap/microk8s/6070/bin:/snap/microk8s/6070/usr/sbin:/snap/microk8s/6070/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin","_=/snap/microk8s/6070/bin/sleep"]}],"id":713,"name":"sched_process_exec","returnValue":0}}],"metadata":{"Version":"1","Description":"LD_PRELOAD usage was detected. LD_PRELOAD lets you load your library before any other library, allowing you to hook functions in a process. Adversaries may use this technique to change your applications' behavior or load their own programs.","Tags":null,"Properties":{"Category":"persistence","Kubernetes_Technique":"","Severity":2,"Technique":"Hijack Execution Flow","external_id":"T1574","id":"attack-pattern--aedfca76-3b30-4866-b2aa-0f1d7fd1e4b6","signatureID":"TRC-107","signatureName":"LD_PRELOAD code injection detected"}}}

And I can list the binaries causing the triggering:

$ microk8s.kubectl logs tracee-zc65w | jq -r '.processName' | sort -u
10-cni-restart
10-cni-restart-
90-calico-apply
bandwidth
calico
calico-ipam
chgrp
chmod
getent
grep
ip6tables
iptables
loopback
null
openssl
portmap
python3
runc
sleep
snapctl

$ microk8s.kubectl logs tracee-jvbtb | jq -r '.processName' | sort -u
10-cni-restart
10-cni-restart-
90-calico-apply
bandwidth
calico
calico-ipam
chgrp
chmod
getent
grep
ip6tables
iptables
jq
loopback
null
openssl
portmap
python3
runc
sleep
snapctl

Microk8s runs using snaps (just like flatpaks) and that could be causing the fause positives (needs to check before confirming), but it could also be that the signature is being triggered by something else.

rafaeldtinoco avatar Nov 03 '23 02:11 rafaeldtinoco

If you will check the signatures you have printed, you can see that all of them are actually true-positives. All of these executions use the LD_LIBRARY_PATH environment variable. Remember that this is not a signature that mark a malicious behavior, but a signature that mark the use of the ld_preload feature.

AlonZivony avatar Nov 05 '23 13:11 AlonZivony

So maybe this issue should be about the signature being useful for the default set or not, and if we should try to ignore some binary paths by default (or binary path sufixes).

Signature is enabled by default but the option it relies on is not. If you change the flag to have the env then the signature might be too verbose to be useful (unless we ignore k8s internal tools ?).

To discuss...

rafaeldtinoco avatar Nov 05 '23 19:11 rafaeldtinoco

I think this event should be low severity and we should filter the default set for signatures with higher severity.

itaysk avatar Nov 11 '23 21:11 itaysk

Removing the LD_LIBRARY_PATH usage will remove most "false positives". It all depends on the goal of the signature. If the goal is to indicate preload logic, then I agree with @itaysk that its should be a low severity signature.

AlonZivony avatar Nov 12 '23 12:11 AlonZivony

Maybe we can add a simple aggregation (in the signature), where the same binary won't be outputted twice? In addition to lowering the severity and/or removing from the default set.

NDStrahilevitz avatar Jan 07 '24 11:01 NDStrahilevitz

I can implement the aggregation mechanism if desired, but what's the verdict on severity? And should we remove this from the default k8s policy?

@yanivagman

NDStrahilevitz avatar Apr 18 '24 08:04 NDStrahilevitz

I can implement the aggregation mechanism if desired, but what's the verdict on severity? And should we remove this from the default k8s policy?

@yanivagman

Yes, we can make it low severity and not emit those kind of "threat detction" events by default. If doing that, we can keep all signatures in the default policy (the user should be able to set the severity of threat events he would like to get)

yanivagman avatar Apr 18 '24 08:04 yanivagman