cilium
cri-o 1.29 Tetragon not show namespace/pod information
What happened?
It seems that with CRI-O 1.29, Tetragon does not visualize information about namespaces and pods.
Check the logs of the test container:
kubectl exec -ti -n tetragon tetragon-r292s -c tetragon -- tetra -d getevents -o compact|grep passwd
time="2024-06-24T14:54:53Z" level=debug msg="Processing event" event="process_exec:{process:{exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzUyOTE2NjY1NDAxMzE6MzU1MzYzNw==\" pid:{value:3553637} uid:{} cwd:\"/\" binary:\"/bin/cat\" arguments:\"/etc/passwd\" flags:\"execve rootcwd clone\" start_time:{seconds:1719240893 nanos:353654638} auid:{value:4294967295} parent_exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjk4NTEwNzk6MzU1Mjc0OQ==\" tid:{value:3553637}} parent:{exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjk4NTEwNzk6MzU1Mjc0OQ==\" pid:{value:3552749} uid:{} cwd:\"/\" binary:\"/bin/sh\" flags:\"execve rootcwd clone\" start_time:{seconds:1719240456 nanos:716961213} auid:{value:4294967295} parent_exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjM4Nzk4MTM6MzU1Mjc0Nw==\" tid:{value:3552749}}} node_name:\"mynode.local\" time:{seconds:1719240893 nanos:353650458}"
🚀 process mynode.local /bin/cat /etc/passwd
time="2024-06-24T14:54:53Z" level=debug msg="Processing event" event="process_exit:{process:{exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzUyOTE2NjY1NDAxMzE6MzU1MzYzNw==\" pid:{value:3553637} uid:{} cwd:\"/\" binary:\"/bin/cat\" arguments:\"/etc/passwd\" flags:\"execve rootcwd clone\" start_time:{seconds:1719240893 nanos:353654638} auid:{value:4294967295} parent_exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjk4NTEwNzk6MzU1Mjc0OQ==\" tid:{value:3553637}} parent:{exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjk4NTEwNzk6MzU1Mjc0OQ==\" pid:{value:3552749} uid:{} cwd:\"/\" binary:\"/bin/sh\" flags:\"execve rootcwd clone\" start_time:{seconds:1719240456 nanos:716961213} auid:{value:4294967295} parent_exec_id:\"ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjI4NzQ4NTUwMjM4Nzk4MTM6MzU1Mjc0Nw==\" tid:{value:3552749}} time:{seconds:1719240893 nanos:354155578}} node_name:\"mynode.local\" time:{seconds:1719240893 nanos:354155575}"
💥 exit mynode.local /bin/cat /etc/passwd 0"
Switching log to tracing mode I can see this messages:
tetragon time="2024-07-05T08:24:38Z" level=trace msg="process_exec: no container ID due to cgroup name not being a compatible ID, ignoring." cgroup.id=98592 cgroup.name=container process.binary=/bin/sh process.exec_id="ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjcyMjM0MTAxNzQ3OTIyOjIyNTA2NA=="
where exec_id is the same from:
{
"process_exec": {
"process": {
"exec_id": "ZGtyLWNiLXV4aTQwMi5pdHRlc3QuY29ybmVyLmxvY2FsOjcyMjM0MTAxNzQ3OTIyOjIyNTA2NA==",
"pid": 225064,
"uid": 0,
"cwd": "/",
"binary": "/bin/sh",
I execute crictl insepct and crictl inspect <container_id>| grep cgroupsPath
and into cgroupsPath:
sudo find /sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/ -type
d
/sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/
/sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/crio-13af5e5e8dd365f35cd40d268140600b80449b3c956c1ee257961ea51dfc1f74
/sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/crio-4ff86f4fe0ebd2556606a6b049b94ae571e3486c4c265230dc8ce87f887ffd15.scope
/sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/crio-4ff86f4fe0ebd2556606a6b049b94ae571e3486c4c265230dc8ce87f887ffd15.scope/container
Tetragon Version
1.1.2
Kernel Version
6.1.0-24
Kubernetes Version
1.29.2
Bugtool
No response
Relevant log output
No response
Anything else?
No response
It seems that this version of cri-o uses paths such as:
/sys/fs/cgroup//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2263b521_f8ac_475e_82a0_95937cce8f0f.slice/crio-4ff86f4fe0ebd2556606a6b049b94ae571e3486c4c265230dc8ce87f887ffd15.scope/container
For the container cgroup that do not work well with tetragon. I think the best solution is to connect to the container runtime (https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto), get the cgroups used, and use the cgroup id to do the mapping. I think we would want to have an option to enable this.
There was a short discussion about this in the Tetragon Community Meeting (July 8th): https://docs.google.com/document/d/1BFMJLdtisiCSLfMct0GHof_ioL-5QVNLEaeMSlk_7Eo/edit#heading=h.cd9xm2lbvnw4.
I'm reopening this to add instructions on how to use the features introduced from #2776
I don't think this is fully solved. I've actually switched over to the new functionality in #2776 and while process execution does get attributed correctly, I'm seeing tcp_connect calls not get attributed to the pod they originated from, and instead coming from the host. Maybe I'm missing something about how this is intended to work though.
I don't think this is fully solved. I've actually switched over to the new functionality in #2776 and while process execution does get attributed correctly, I'm seeing tcp_connect calls not get attributed to the pod they originated from, and instead coming from the host. Maybe I'm missing something about how this is intended to work though.
Can you provide more details (i.e., specific steps on how to reproduce this)? We also recently merged https://github.com/cilium/tetragon/pull/3382 which makes it easier to deploy this configuration.
Yeah! Sorry, I neglected that originally.
So, I can't actually use the helm chart as y'all have provided it, because there hasn't been a release yet. Technically I could pull it down and manually deploy, but there are some complications there. Also, there's actually an error in the merged PR that you can see at https://github.com/cilium/tetragon/blob/af571d8f81fe2f4fe02a0298300cd52baaafefc2/install/kubernetes/tetragon/templates/tetragon_configmap.yaml#L84-L85 (Values.cri.enabled vs Values.cri.enable), which I meant to put a PR up for.
In my case, we're doing post-rendering patching of the existing 1.3.0 chart output because of this. We're using Flux to deploy, so it has a kustomize wrapper around helm that we can use to do dynamic patching, that's what the syntax you're about to see is.
postRenderers:
- kustomize:
patches:
- patch: |
- op: add
path: /data/enable-cri
value: "true"
- op: add
path: /data/cri-endpoint
value: "unix:///var/run/crio/crio.sock"
- op: add
path: /data/enable-cgidmap
value: "true"
target:
kind: ConfigMap
name: tetragon-config
version: v1
- patch: |
- op: add
path: /spec/template/spec/containers/1/volumeMounts/-
value:
mountPath: "/var/run/crio/crio.sock"
name: cri-socket
- op: add
path: /spec/template/spec/volumes/-
value:
name: cri-socket
hostPath:
path: "/var/run/crio/crio.sock"
type: Socket
target:
kind: DaemonSet
name: tetragon
version: v1
group: apps
So, as you can see here, we're patching in the relevant values in the ConfigMap. And then we're also patching the DaemonSet to mount in the CRI-O socket. Since the 1.3.0 chart doesn't have this functionality we've had to do it this way for now. As far as I can tell from reading the chart and the changes, this appears to be correct configuration. We do see mapping of syscalls correctly attributed to the originating pod. However, when looking at TCP connections, it's always attributed to the node, regardless of the network namespace the connection originated from.
Yeah! Sorry, I neglected that originally.
So, I can't actually use the helm chart as y'all have provided it, because there hasn't been a release yet. Technically I could pull it down and manually deploy, but there are some complications there. Also, there's actually an error in the merged PR that you can see at
tetragon/install/kubernetes/tetragon/templates/tetragon_configmap.yaml
Lines 84 to 85 in af571d8 enable-cri: {{ .Values.tetragon.cri.enabled | quote }} {{- if and (.Values.tetragon.cri.enable) (.Values.tetragon.cri.socketHostPath) }} (Values.cri.enabled vs Values.cri.enable), which I meant to put a PR up for.
Ah, thanks. Here's a PR to fix it: https://github.com/cilium/tetragon/pull/3499
In my case, we're doing post-rendering patching of the existing 1.3.0 chart output because of this. We're using Flux to deploy, so it has a kustomize wrapper around helm that we can use to do dynamic patching, that's what the syntax you're about to see is.
postRenderers: - kustomize: patches: - patch: | - op: add path: /data/enable-cri value: "true" - op: add path: /data/cri-endpoint value: "unix:///var/run/crio/crio.sock" - op: add path: /data/enable-cgidmap value: "true" target: kind: ConfigMap name: tetragon-config version: v1 - patch: | - op: add path: /spec/template/spec/containers/1/volumeMounts/- value: mountPath: "/var/run/crio/crio.sock" name: cri-socket - op: add path: /spec/template/spec/volumes/- value: name: cri-socket hostPath: path: "/var/run/crio/crio.sock" type: Socket target: kind: DaemonSet name: tetragon version: v1 group: appsSo, as you can see here, we're patching in the relevant values in the ConfigMap. And then we're also patching the DaemonSet to mount in the CRI-O socket. Since the 1.3.0 chart doesn't have this functionality we've had to do it this way for now. As far as I can tell from reading the chart and the changes, this appears to be correct configuration. We do see mapping of syscalls correctly attributed to the originating pod. However, when looking at TCP connections, it's always attributed to the node, regardless of the network namespace the connection originated from.
What is the version of tetragon (i.e., the container) you are using? Are you using the 1.3.0 version?
Yes, 1.3.0, sha256:46a81dc94adea0b2748931b261cf537fdd1c3635c76ebed6da90186650709712
Can you try the latest image(s)? You can find them at https://github.com/cilium/tetragon/actions/runs/13837643520 Also, when you say network connections are you using a specific policy?
I'll see what I can do on testing the images. I'm going to plead a little bit of ignorance here on the policy thing as I'm not clear on if the policy is involved, but I'll happily paste one that would be related if it is at the end of this post. I was watching tetra getevents to see this, and it's not clear to me if that only logs if there's a related policy. I can see connect events occurring, but it's always the node name rather than a pod name showing up. As I was typing this, I realized I could probably pull an example. I checked a cluster and found that it actually was working, and now I'm realizing the difference between the two clusters. On our legacy clusters, which run Calico, the attribution isn't working. On our modern clusters, which run Cilium, it does. Maybe there's something about Calico or our particular configuration that inhibits the attribution? Below is the policy we're using on the clusters.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: external-tcp
spec:
kprobes:
- args:
- index: 0
maxData: false
returnCopy: false
type: sock
call: tcp_connect
return: false
selectors:
- matchActions:
- action: Post
rateLimit: 10m
matchArgs:
- index: 0
operator: NotDAddr
values:
- 127.0.0.1
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 127.0.0.0/8
- 169.254.169.254
- 169.254.20.10
syscall: false
