tetragon icon indicating copy to clipboard operation
tetragon copied to clipboard

container metadata is missing from events if the policy is matched during kubectl exec

Open holyspectral opened this issue 4 months ago • 2 comments

What happened? How can we reproduce this?

  1. In a kubernetes environment (verified on kind + Ubuntu 24.04 and EKS), deploy tetragon.
  2. Deploy the tracing policy below, which blocks all executables matching the k8s pod label app: ubuntu
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: deploy-ubuntu-deployment
spec:
  kprobes:
  - args:
    - index: 0
      type: linux_binprm
    call: security_bprm_creds_for_exec
    selectors:
    - matchActions:
      - action: Override
        argError: -1
    syscall: false
  options:
  - name: disable-kprobe-multi
    value: "1"
  podSelector:
    matchLabels:
      app: ubuntu
  1. Run a program in a pod that matches app: ubuntu using kubectl exec.
  2. The executable is blocked correctly, but its event doesn't have any information associated with the pod, as below:

kubectl-exec.json

  1. If I remove the override part from the tracing policy, the program can run without problem, as expected. However, its event still doesn't come with its container metadata.
  2. If I run the program not through kubectl exec directly, for example, from a shell that I've established before, the event comes with container metadata correctly.

normal.json

Tetragon Version

/ # tetra version
CLI version: v1.5.0

Kernel Version

/ # tetra version CLI version: v1.5.0

(EKS)

/ # uname -a
Linux ip-xxxxxx.compute.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep  9 12:21:26 UTC 2025 x86_64 Linux

(kind)

/ # uname -a
Linux kind-control-plane 6.14.0-32-generic #32~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Sep  2 14:21:04 UTC 2 x86_64 Linux

Kubernetes Version

(EKS)

$ kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.32.9-eks-113cf36

(kind)

$ kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.33.1

Bugtool

This command runs on EKS:

/ # tetra bugtool
level=info msg="saving init info"
level=info msg="retrieving lib directory" libDir=/var/lib/tetragon
level=warn msg="not an object file, ignoring" path=/var/lib/tetragon
level=warn msg="no btf filename in tetragon config, attempting to fall back to /sys/kernel/btf/vmlinux"
level=info msg="btf file added" btfFname=/sys/kernel/btf/vmlinux
level=info msg="tetragon log file added" exportFname=/var/run/cilium/tetragon/tetragon.log
level=info msg="contacting metrics server" metricsAddr=http://localhost:2112/metrics
level=info msg="executed command" cmd=/bin/dmesg ret=0 dstFname=dmesg.out
level=info msg="executed command" cmd="/sbin/tc filter show dev lo ingress" ret=0 dstFname=tc-info.lo.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev lo egress" ret=0 dstFname=tc-info.lo.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens5 ingress" ret=0 dstFname=tc-info.ens5.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens5 egress" ret=0 dstFname=tc-info.ens5.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enif20915368b5 ingress" ret=0 dstFname=tc-info.enif20915368b5.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enif20915368b5 egress" ret=0 dstFname=tc-info.enif20915368b5.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enieddc18b2fc6 ingress" ret=0 dstFname=tc-info.enieddc18b2fc6.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enieddc18b2fc6 egress" ret=0 dstFname=tc-info.enieddc18b2fc6.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens6 ingress" ret=0 dstFname=tc-info.ens6.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev ens6 egress" ret=0 dstFname=tc-info.ens6.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni43ddcaf029e ingress" ret=0 dstFname=tc-info.eni43ddcaf029e.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni43ddcaf029e egress" ret=0 dstFname=tc-info.eni43ddcaf029e.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enifaffa0817c2 ingress" ret=0 dstFname=tc-info.enifaffa0817c2.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enifaffa0817c2 egress" ret=0 dstFname=tc-info.enifaffa0817c2.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni3fbdac6a2ec ingress" ret=0 dstFname=tc-info.eni3fbdac6a2ec.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni3fbdac6a2ec egress" ret=0 dstFname=tc-info.eni3fbdac6a2ec.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni85d6be82070 ingress" ret=0 dstFname=tc-info.eni85d6be82070.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev eni85d6be82070 egress" ret=0 dstFname=tc-info.eni85d6be82070.egress
level=info msg="executed command" cmd="/sbin/tc filter show dev enid197514c16a ingress" ret=0 dstFname=tc-info.enid197514c16a.ingress
level=info msg="executed command" cmd="/sbin/tc filter show dev enid197514c16a egress" ret=0 dstFname=tc-info.enid197514c16a.egress
level=info msg="executed command" cmd="/usr/bin/bpftool map show -j" ret=0 dstFname=bpftool-maps.json
level=info msg="executed command" cmd="/usr/bin/bpftool prog show -j" ret=0 dstFname=bpftool-progs.json
level=info msg="executed command" cmd="/usr/bin/bpftool cgroup tree -j" ret=0 dstFname=bpftool-cgroups.json
level=info msg="Dumping gops information" gops-address=localhost:8118 gops-path=/usr/bin/gops
level=info msg="executed command" cmd="/usr/bin/gops stack localhost:8118" ret=0 dstFname=gops.stack
level=info msg="executed command" cmd="/usr/bin/gops stats localhost:8118" ret=0 dstFname=gops.stats
level=info msg="executed command" cmd="/usr/bin/gops memstats localhost:8118" ret=0 dstFname=gops.memstats
level=info msg="Contacting gops server for pprof dump" gops-address=localhost:8118
level=info msg="Successfully dumped gops pprof" gops-address=localhost:8118 gops-path=/usr/bin/gops profile=cpu
level=info msg="Contacting gops server for pprof dump" gops-address=localhost:8118
level=info msg="Successfully dumped gops pprof" gops-address=localhost:8118 gops-path=/usr/bin/gops profile=heap
level=info msg="dumped tracing policies in tracing-policies.json"
level=info msg="executed command" cmd="/usr/bin/pmap -x 1" ret=0 dstFname=pmap.out
level=info msg="cgroup file added" file=memory.current
level=info msg="cgroup file added" file=memory.stat
level=info msg="BPF maps checks added" file=debugmaps.json
level=info msg="executed command" cmd="/bin/cat /sys/kernel/tracing/trace" ret="exit status 1" dstFname=trace

Relevant log output


Anything else?

No response

holyspectral avatar Oct 07 '25 21:10 holyspectral

If it's okay, we would be happy to contribute and fix this issue.

holyspectral avatar Oct 07 '25 21:10 holyspectral

If it's okay, we would be happy to contribute and fix this issue.

sure feel free to investigate and find out what's the cause of this! Thank you

mtardy avatar Oct 08 '25 09:10 mtardy