Kernel version 5.18.2-1 not supported

[jjsluck0907]

os： CentOS Linux release 7.9.2009 problem：After upgrading the latest kernel with elrepo, the installation of tetragon prompts that the kernel version is not supported

log： time="2022-06-07T08:10:36Z" level=info msg="Loaded config from directory" config-dir=/etc/tetragon time="2022-06-07T08:10:36Z" level=info msg="Starting tetragon" version=v0.8.0 time="2022-06-07T08:10:36Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: cilium-bpf: config-dir:/etc/tetragon config-file: debug:false enable-cilium-api:false enable-export-aggregation:false enable-k8s-api:true enable-process-ancestors:true enable-process-cred:false enable-process-ns:false export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{"event_set":["PROCESS_EXEC", "PROCESS_EXIT", "PROCESS_KPROBE"]} export-denylist:{"health_check":true}\n{"namespace":["", "cilium", "kube-system"]} export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-rotation-interval:0s export-filename:/var/run/cilium/tetragon/tetragon.log export-rate-limit:-1 force-small-progs:false ignore-missing-progs:false kernel: log-format:text log-level:info metrics-server::2112 netns-dir:/var/run/docker/netns/ process-cache-size:65536 procfs:/procRoot run-standalone:false server-address:localhost:54321 verbose:0]" time="2022-06-07T08:10:36Z" level=info msg="Available sensors" sensors= time="2022-06-07T08:10:36Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor" time="2022-06-07T08:10:36Z" level=info msg="Registered probe types" types="kprobe sensor, tracepoint sensor" time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/sys/kernel/btf/vmlinux time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/var/lib/tetragon/metadata/vmlinux-5.18.2-1.el7.elrepo.x86_64 time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/var/lib/tetragon/btf time="2022-06-07T08:10:36Z" level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: Kernel version "5.18.2-1.el7.elrepo.x86_64" BTF search failed kernel is

[willfindlay]

You need to compile your kernel with BTF info. Check for the existence of /sys/kernel/btf.

[willfindlay]

Alternatively, you can generate the BTF yourself and load it into the Tetragon pod but that's a bit more of an advanced use case.

[nisainan]

Is there any demo or guide of fixing this problem?

[jjsluck0907]

No sys/kernel/btf。Is there any relevant documentation? Thank you

[willfindlay]

https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-generation This explains how to generate the BTF manually. You could then set the helm values such that this is mounted into the Tetragon container and used by Tetragon.

Alternatively, you'll need to recompile your kernel with BTF support. That involves setting some kconfig flags and compiling it with a recent version of pahole installed on your system.

[willfindlay]

If you end up going the first route, here are the helm values you're looking for:

https://github.com/cilium/tetragon/blob/402d16f084669bccf1562dc4d2fda8554c0f3715/install/kubernetes/values.yaml#L14

https://github.com/cilium/tetragon/blob/402d16f084669bccf1562dc4d2fda8554c0f3715/install/kubernetes/values.yaml#L118

[jjsluck0907]

BTF is enabled by default using cenots 8 。 Do not upgrade the latest kernel. The default kernel version will not be reported as unsupported kernel when installing tetragon。 Upgrade the latest kernel version that is not supported by the same message。 https://github.com/aquasecurity/btfhub/blob/main/docs/supported-distros.md

[Forsworns]

If you end up going the first route, here are the helm values you're looking for:

https://github.com/cilium/tetragon/blob/402d16f084669bccf1562dc4d2fda8554c0f3715/install/kubernetes/values.yaml#L14

https://github.com/cilium/tetragon/blob/402d16f084669bccf1562dc4d2fda8554c0f3715/install/kubernetes/values.yaml#L118

I download a pre-built btf file from BTF-Hub and place it to /var/lib/tetragon/. Then I read the files in install/kubernetes/templates and change the install/kubernetes/values.yaml to

extraHostPathMounts: 
- name: "btf-dir"
  mountPath: /var/lib/tetragon/
  mountPropagation: HostToContainer
  readOnly: false
  ...
  btf: "/var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf"

However, the file is not detected and tetragon reports the btf file does not exists.

Here are related logs:

> ls /var/lib/tetragon/

4.18.0-147.5.1.el8_1.x86_64.btf 

> kubectl logs -f "$(kubectl get pod -n kube-system | grep tetragon | awk '{print $1}')" -n kube-system -c tetragon

time="2022-06-20T02:07:35Z" level=warning msg="BPF filesystem is going to be mounted automatically in /run/cilium/bpffs. However, it probably means that Cilium is running inside container and BPFFS is not mounted on the host. for more information, see: https://cilium.link/err-bpf-mount"
time="2022-06-20T02:07:35Z" level=info msg="Available sensors" sensors=
time="2022-06-20T02:07:35Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor"
time="2022-06-20T02:07:35Z" level=info msg="Registered probe types" types="tracepoint sensor, kprobe sensor"
**time="2022-06-20T02:07:35Z" level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: User specified BTF does not exist: stat /var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf: no such file or directory"**

> kubectl describe pod "$(kubectl get pod -n kube-system | grep tetragon | awk '{print $1}')" -n kube-system
tetragon:
   Mounts:
      /etc/tetragon from tetragon-config (ro)
      /procRoot from host-proc (rw)
      /sys/fs/bpf from bpf-maps (rw)
      /var/lib/tetragon/ from btf-dir (rw)
      /var/run/cilium from cilium-run (rw)
      /var/run/cilium/tetragon from export-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c462t (ro)

[kkourt]

However, the file is not detected and tetragon reports the btf file does not exists.

Does the file var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf exist on the host or in the tetragon pod? If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

[Forsworns]

If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

Sure, the file is not in the pod. But is it expected?

The extraHostPathMounts has mounted the host path /var/lib/tetragon/ to the tetragon pod. I thought the btf file should be shared in this way.

Did I misunderstand the effects of extraHostPathMounts?

[kkourt]

If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

Sure, the file is not in the pod. But is it expected?

The extraHostPathMounts has mounted the host path /var/lib/tetragon/ to the tetragon pod. I thought the btf file should be shared in this way.

Did I misunderstand the effects of extraHostPathMounts?

Ah right! I think you are correct!

In that case, I'm not sure why we are getting that error if the file exists: https://github.com/cilium/tetragon/blob/1b7d2315df37b8f88c13e1d15dfdb62f7d425629/pkg/btf/btf.go#L86-L89

https://github.com/cilium/tetragon/blob/1b7d2315df37b8f88c13e1d15dfdb62f7d425629/pkg/btf/btf.go#L72-L74

https://github.com/cilium/tetragon/blob/1b7d2315df37b8f88c13e1d15dfdb62f7d425629/pkg/btf/btf.go#L25-L27

Would it be possible to kubectl exec into the tetragon pod and check that the file is indeed there?

[kkourt]

Could it be that there the btf value is not properly passed?

At the beginning of the tetragon logs, there should be a line that prints the full configuration: https://github.com/cilium/tetragon/blob/1b7d2315df37b8f88c13e1d15dfdb62f7d425629/cmd/tetragon/main.go#L107-L108

Could you please check it out?

[willfindlay]

Is this a KinD cluster by any chance? If so you would need to make sure the file is present in the KinD container in order for extraHostPathMounts to work properly.

[Forsworns]

Is this a KinD cluster by any chance? If so you would need to make sure the file is present in the KinD container in order for extraHostPathMounts to work properly.

Yes, it'a local cluster created via KinD. Thanks, I understand it now. The host path is in fact the KinD container in this situation

[Forsworns]

Could it be that there the btf value is not properly passed?

Yes, they are correctly set. Tetragon works well :) I omitted it in the above comments, but I remembered it.