tetragon
tetragon copied to clipboard
container metadata is missing from events if the policy is matched during kubectl exec
What happened? How can we reproduce this?
- In a kubernetes environment (verified on kind + Ubuntu 24.04 and EKS), deploy tetragon.
- Deploy the tracing policy below, which blocks all executables matching the k8s pod label
app: ubuntu
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: deploy-ubuntu-deployment
spec:
kprobes:
- args:
- index: 0
type: linux_binprm
call: security_bprm_creds_for_exec
selectors:
- matchActions:
- action: Override
argError: -1
syscall: false
options:
- name: disable-kprobe-multi
value: "1"
podSelector:
matchLabels:
app: ubuntu
- Run a program in a pod that matches
app: ubuntuusingkubectl exec. - The executable is blocked correctly, but its event doesn't have any information associated with the pod, as below:
- If I remove the override part from the tracing policy, the program can run without problem, as expected. However, its event still doesn't come with its container metadata.
- If I run the program not through
kubectl execdirectly, for example, from a shell that I've established before, the event comes with container metadata correctly.
Tetragon Version
/ # tetra version
CLI version: v1.5.0
Kernel Version
/ # tetra version CLI version: v1.5.0
(EKS)
/ # uname -a
Linux ip-xxxxxx.compute.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 Linux
(kind)
/ # uname -a
Linux kind-control-plane 6.14.0-32-generic #32~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Sep 2 14:21:04 UTC 2 x86_64 Linux
Kubernetes Version
(EKS)
$ kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.32.9-eks-113cf36
(kind)
$ kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.33.1
Bugtool
This command runs on EKS:
/ # tetra bugtool
level=info msg="saving init info"
level=info msg="retrieving lib directory" libDir=/var/lib/tetragon
level=warn msg="not an object file, ignoring" path=/var/lib/tetragon
level=warn msg="no btf filename in tetragon config, attempting to fall back to /sys/kernel/btf/vmlinux"
level=info msg="btf file added" btfFname=/sys/kernel/btf/vmlinux
level=info msg="tetragon log file added" exportFname=/var/run/cilium/tetragon/tetragon.log
level=info msg="contacting metrics server" metricsAddr=http://localhost:2112/metrics
level=info msg="executed command" cmd=/bin/dmesg ret=0 dstFname=dmesg.out
level=info msg="executed command" cmd="/sbin/tc filter show dev lo ingress" ret=0 dstFname=tc-info.lo.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev lo egress" ret=0 dstFname=tc-info.lo.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens5 ingress" ret=0 dstFname=tc-info.ens5.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens5 egress" ret=0 dstFname=tc-info.ens5.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enif20915368b5 ingress" ret=0 dstFname=tc-info.enif20915368b5.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enif20915368b5 egress" ret=0 dstFname=tc-info.enif20915368b5.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enieddc18b2fc6 ingress" ret=0 dstFname=tc-info.enieddc18b2fc6.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enieddc18b2fc6 egress" ret=0 dstFname=tc-info.enieddc18b2fc6.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens6 ingress" ret=0 dstFname=tc-info.ens6.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens6 egress" ret=0 dstFname=tc-info.ens6.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni43ddcaf029e ingress" ret=0 dstFname=tc-info.eni43ddcaf029e.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni43ddcaf029e egress" ret=0 dstFname=tc-info.eni43ddcaf029e.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enifaffa0817c2 ingress" ret=0 dstFname=tc-info.enifaffa0817c2.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enifaffa0817c2 egress" ret=0 dstFname=tc-info.enifaffa0817c2.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni3fbdac6a2ec ingress" ret=0 dstFname=tc-info.eni3fbdac6a2ec.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni3fbdac6a2ec egress" ret=0 dstFname=tc-info.eni3fbdac6a2ec.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni85d6be82070 ingress" ret=0 dstFname=tc-info.eni85d6be82070.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni85d6be82070 egress" ret=0 dstFname=tc-info.eni85d6be82070.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enid197514c16a ingress" ret=0 dstFname=tc-info.enid197514c16a.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enid197514c16a egress" ret=0 dstFname=tc-info.enid197514c16a.egress
level=info msg="executed command" cmd="/usr/bin/bpftool map show -j" ret=0 dstFname=bpftool-maps.json
level=info msg="executed command" cmd="/usr/bin/bpftool prog show -j" ret=0 dstFname=bpftool-progs.json
level=info msg="executed command" cmd="/usr/bin/bpftool cgroup tree -j" ret=0 dstFname=bpftool-cgroups.json
level=info msg="Dumping gops information" gops-address=localhost:8118 gops-path=/usr/bin/gops
level=info msg="executed command" cmd="/usr/bin/gops stack localhost:8118" ret=0 dstFname=gops.stack
level=info msg="executed command" cmd="/usr/bin/gops stats localhost:8118" ret=0 dstFname=gops.stats
level=info msg="executed command" cmd="/usr/bin/gops memstats localhost:8118" ret=0 dstFname=gops.memstats
level=info msg="Contacting gops server for pprof dump" gops-address=localhost:8118
level=info msg="Successfully dumped gops pprof" gops-address=localhost:8118 gops-path=/usr/bin/gops profile=cpu
level=info msg="Contacting gops server for pprof dump" gops-address=localhost:8118
level=info msg="Successfully dumped gops pprof" gops-address=localhost:8118 gops-path=/usr/bin/gops profile=heap
level=info msg="dumped tracing policies in tracing-policies.json"
level=info msg="executed command" cmd="/usr/bin/pmap -x 1" ret=0 dstFname=pmap.out
level=info msg="cgroup file added" file=memory.current
level=info msg="cgroup file added" file=memory.stat
level=info msg="BPF maps checks added" file=debugmaps.json
level=info msg="executed command" cmd="/bin/cat /sys/kernel/tracing/trace" ret="exit status 1" dstFname=trace
Relevant log output
Anything else?
No response
If it's okay, we would be happy to contribute and fix this issue.
If it's okay, we would be happy to contribute and fix this issue.
sure feel free to investigate and find out what's the cause of this! Thank you