tetragon icon indicating copy to clipboard operation
tetragon copied to clipboard
verified publisher icon cilium

Upgrading from Tetragon v1.3.0 to v1.4.0 results in a validation failed: validateKprobeSpec: ksyms.KernelSymbols: no symbols found error

Open quentinkhoo opened this issue 8 months ago • 2 comments

What happened?

On an AWS EC2 Instance running a ubuntu-22.04 server AMI, i tried upgrading tetragon from v1.3.0 to v1.4.0. The policies that were in /etc/tetragon/tetragon.tp.d were always working on v1.3.0 but when upgrading (by running uninstall.sh and install.sh) to v1.4.0, tetragon just stopped and threw an error as shown below:

Jun 10 08:07:14 ip-10-55-0-100 tetragon[108350]: time="2025-06-10T08:07:14Z" level=fatal msg="Failed to start tetragon" error="policy handler 'tracing' failed loading policy 'create-below-dev-directory': validation failed: validateKprobeSpec: ksyms.KernelSymbols: no symbols found"

Tetragon Version

version 1.4.0

Kernel Version

Linux ip-10-210-0-45 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 20:59:24 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Kubernetes Version

NA

Bugtool

$ sudo tetra bugtool
time="2025-06-10T10:01:04Z" level=info msg="saving init info"
time="2025-06-10T10:01:04Z" level=info msg="retrieving lib directory" libDir=/usr/local/lib/tetragon/bpf/
time="2025-06-10T10:01:04Z" level=warning msg="not an object file, ignoring" path=/usr/local/lib/tetragon/bpf/
time="2025-06-10T10:01:07Z" level=warning msg="no btf filename in tetragon config, attempting to fall back to /sys/kernel/btf/vmlinux"
time="2025-06-10T10:01:07Z" level=info msg="btf file added" btfFname=/sys/kernel/btf/vmlinux
time="2025-06-10T10:01:07Z" level=info msg="tetragon log file added" exportFname=/var/log/tetragon/tetragon.log
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd=/usr/bin/dmesg dstFname=dmesg.out ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/sbin/tc filter show dev lo ingress" dstFname=tc-info.lo.ingress ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/sbin/tc filter show dev lo egress" dstFname=tc-info.lo.egress ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/sbin/tc filter show dev ens5 ingress" dstFname=tc-info.ens5.ingress ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/sbin/tc filter show dev ens5 egress" dstFname=tc-info.ens5.egress ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/local/lib/tetragon/bpftool map show -j" dstFname=bpftool-maps.json ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/local/lib/tetragon/bpftool prog show -j" dstFname=bpftool-progs.json ret=0
time="2025-06-10T10:01:07Z" level=info msg="executed command" cmd="/usr/local/lib/tetragon/bpftool cgroup tree -j" dstFname=bpftool-cgroups.json ret="exit status 255"
time="2025-06-10T10:01:07Z" level=info msg="Skipping gops dump info as daemon is running without gops, use --gops-address to enable gops"
time="2025-06-10T10:01:07Z" level=warning msg="failed to open policyfilter map" error="no such file or directory"
time="2025-06-10T10:01:08Z" level=warning msg="failed to list tracing policies: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 3336\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html;charset=utf-8\\\\r\\\\nDate: Tue, 10 Jun 2025 10:01:08 GMT\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nServer: squid/5.9\\\\r\\\\nVary: Accept-Language\\\\r\\\\nVia: 1.1 ip-10-55-0-100 (squid/5.9)\\\\r\\\\nX-Cache: MISS from ip-10-55-0-100\\\\r\\\\nX-Cache-Lookup: NONE from ip-10-55-0-100:3128\\\\r\\\\nX-Squid-Error: ERR_PROTOCOL_UNKNOWN 0\\\\r\\\\n\\\\r\\\\n<!DOCTYPE html PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01//EN\\\\\\\" \\\\\\\"http://www.w3.org/TR/html4/strict.dtd\\\\\\\">\\\\n<html><head>\\\\n<meta type=\\\\\\\"copyright\\\\\\\" content=\\\\\\\"Copyright (C) 1996-2020 The Squid Software Foundation and contributors\\\\\\\">\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=utf-8\\\\\\\">\\\\n<title>ERROR: The requested URL could not be retrieved</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\"><!-- \\\\n /*\\\\n * Copyright (C) 1996-2023 The Squid Software Foundation and contributors\\\\n *\\\\n * Squid software is distributed under GPLv2+ license and includes\\\\n * contributions from numerous individuals and organizations.\\\\n * Please see the COPYING and CONTRIBUTORS files for details.\\\\n */\\\\n\\\\n/*\\\\n Stylesheet for Squid Error pages\\\\n Adapted from design by Free CSS Templates\\\\n http://www.freecsstemplates.org\\\\n Released for free under a Creative Commons Attribution 2.5 License\\\\n*/\\\\n\\\\n/* Page basics */\\\\n* {\\\\n\\\\tfont-family: verdana, sans-serif;\\\\n}\\\\n\\\\nhtml body {\\\\n\\\\tmargin: 0;\\\\n\\\\tpadding: 0;\\\\n\\\\tbackground: #efefef;\\\\n\\\\tfont-size: 12px;\\\\n\\\\tcolor: #1e1e1e;\\\\n}\\\\n\\\\n/* Page displayed title area */\\\\n#titles {\\\\n\\\\tmargin-left: 15px;\\\\n\\\\tpadding: 10px;\\\\n\\\\tpadding-left: 100px;\\\\n\\\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\\\n}\\\\n\\\\n/* initial title */\\\\n#titles h1 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n#titles h2 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n\\\\n/* special event: FTP success page titles */\\\\n#titles ftpsuccess {\\\\n\\\\tbackground-color:#00ff00;\\\\n\\\\twidth:100%;\\\\n}\\\\n\\\\n/* Page displayed body content area */\\\\n#content {\\\\n\\\\tpadding: 10px;\\\\n\\\\tbackground: #ffffff;\\\\n}\\\\n\\\\n/* General text */\\\\np {\\\\n}\\\\n\\\\n/* error brief description */\\\\n#error p {\\\\n}\\\\n\\\\n/* some data which may have caused the problem */\\\\n#data {\\\\n}\\\\n\\\\n/* the error message received from the system or other software */\\\\n#sysmsg {\\\\n}\\\\n\\\\npre {\\\\n}\\\\n\\\\n/* special event: FTP / Gopher directory listing */\\\\n#dirmsg {\\\\n    font-family: courier, monospace;\\\\n    color: black;\\\\n    font-size: 10pt;\\\\n}\\\\n#dirlisting {\\\\n    margin-left: 2%;\\\\n    margin-right: 2%;\\\\n}\\\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\\\n    border-bottom: groove;\\\\n}\\\\n#dirlisting td.size {\\\\n    width: 50px;\\\\n    text-align: right;\\\\n    padding-right: 5px;\\\\n}\\\\n\\\\n/* horizontal lines */\\\\nhr {\\\\n\\\\tmargin: 0;\\\\n}\\\\n\\\\n/* page displayed footer area */\\\\n#footer {\\\\n\\\\tfont-size: 9px;\\\\n\\\\tpadding-left: 10px;\\\\n}\\\\n\\\\n\\\\nbody\\\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\\\n:lang(he) { direction: rtl; }\\\\n --></style>\\\\n</head><body id=ERR_PROTOCOL_UNKNOWN>\\\\n<div id=\\\\\\\"titles\\\\\\\">\\\\n<h1>ERROR</h1>\\\\n<h2>The requested URL could not be retrieved</h2>\\\\n</div>\\\\n<hr>\\\\n\\\\n<div id=\\\\\\\"content\\\\\\\">\\\\n<p>The following error was encountered while trying to retrieve the URL: <a href=\\\\\\\"error:invalid-request\\\\\\\">error:invalid-request</a></p>\\\\n\\\\n<blockquote id=\\\\\\\"error\\\\\\\">\\\\n<p><b>Unsupported Protocol</b></p>\\\\n</blockquote>\\\\n\\\\n<p>Squid does not support some access protocols. For example, the SSH protocol is currently not supported.</p>\\\\n\\\\n<p>Your cache administrator is <a href=\\\\\\\"mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_PROTOCOL_UNKNOWN&amp;body=CacheHost%3A%20ip-10-55-0-100%0D%0AErrPage%3A%20ERR_PROTOCOL_UNKNOWN%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Tue,%2010%20Jun%202025%2010%3A01%3A08%20GMT%0D%0A%0D%0AClientIP%3A%20127.0.0.1%0D%0A%0D%0AHTTP%20Request%3A%0D%0A%0D%0A%0D%0A\\\\\\\">webmaster</a>.</p>\\\\n<br>\\\\n</div>\\\\n\\\\n<hr>\\\\n<div id=\\\\\\\"footer\\\\\\\">\\\\n<p>Generated Tue, 10 Jun 2025 10:01:08 GMT by ip-10-55-0-100 (squid/5.9)</p>\\\\n<!-- ERR_PROTOCOL_UNKNOWN -->\\\\n</div>\\\\n</body></html>\\\\n\\\"\""
time="2025-06-10T10:01:08Z" level=info msg="executed command" cmd="/usr/bin/pmap -x 123265" dstFname=pmap.out ret="exit status 42"
time="2025-06-10T10:01:08Z" level=info msg="cgroup file added" file=memory.current
time="2025-06-10T10:01:08Z" level=info msg="cgroup file added" file=memory.stat
time="2025-06-10T10:01:08Z" level=warning msg="failed to run BPF maps checks" error="make sure tetragon is running and you have enough permissions: stat /sys/fs/bpf/tetragon: no such file or directory"
time="2025-06-10T10:01:08Z" level=info msg="executed command" cmd="/usr/bin/cat /sys/kernel/tracing/trace" dstFname=trace ret=0

Relevant log output

Jun 09 08:56:33 ip-10-55-0-100 systemd[1]: Started Tetragon eBPF-based Security Observability and Runtime Enforcement.
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="Starting tetragon" version=v1.4.0
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="config settings" config="map[bpf-dir:tetragon bpf-lib:/usr/local/lib/tetragon/bpf/ btf: cgroup-rate: cluster-name: config-dir: cpuprofile: cri-endpoint: data-cache-size:1024 debug:false disable-kprobe-multi:false enable-cgidmap:false enable-cgidmap-debug:false enable-cgtrackerid:true enable-compatibility-syscall64-size-type:false enable-cri:false enable-export-aggregation:false enable-k8s-api:false enable-msg-handling-latency:false enable-pid-set-filter:false enable-pod-info:false enable-policy-filter:false enable-policy-filter-cgroup-map:false enable-policy-filter-debug:false enable-process-ancestors:false enable-process-cred:false enable-process-kprobe-ancestors:false enable-process-lsm-ancestors:false enable-process-ns:false enable-process-tracepoint-ancestors:false enable-process-uprobe-ancestors:false enable-tracing-policy-crd:true event-cache-retries:15 event-cache-retry-delay:2 event-queue-size:10000 execve-map-entries:0 execve-map-size: export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{\n  \"event_set\": [\n    \"PROCESS_KPROBE\",\n    \"PROCESS_UPROBE\",\n    \"PROCESS_TRACEPOINT\",\n    \"PROCESS_LSM\"\n  ]\n} export-denylist: export-file-compress:true export-file-max-backups:5 export-file-max-size-mb:10 export-file-perm:600 export-file-rotation-interval:0s export-filename:/var/log/tetragon/tetragon.log export-rate-limit:-1 expose-stack-addresses:false field-filters: force-large-progs:false force-small-progs:false generate-docs:false gops-address: health-server-address::6789 health-server-interval:10 k8s-kubeconfig-path: keep-sensors-on-exit:false kernel: log-format:text log-level:info memprofile: metrics-label-filter:namespace,workload,pod,binary metrics-server: netns-dir:/var/run/docker/netns/ pprof-address: process-cache-gc-interval:30s process-cache-size:65536 procfs:/proc/ rb-queue-size:65535 rb-size:0 rb-size-total:0 redaction-filters: release-pinned-bpf:true server-address:unix:///var/run/tetragon/tetragon.sock tracing-policy: tracing-policy-dir:/etc/tetragon/tetragon.tp.d username-metadata:disabled verbose:0]"
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="Tetragon current security context" AppArmor=unconfined Lockdown=none SELinux=unconfined Smack=
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="Tetragon pid file creation succeeded" pid=224406 pidfile=/var/run/tetragon/tetragon.pid
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="BPF: successfully released pinned BPF programs and maps" bpf-dir=/sys/fs/bpf/tetragon
Jun 09 08:56:33 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:33Z" level=info msg="BTF discovery: default kernel btf file found" btf-file=/sys/kernel/btf/vmlinux
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="BPF detected features: override_return: true, buildid: true, kprobe_multi: false, uprobe_multi false, fmodret: true, fmodret_syscall: true, signal: true, large: true, link_pin: true, lsm: false, missed_stats_kprobe_multi: true, missed_stats_kprobe: true"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Cgroup mode detection succeeded" cgroup.fs=/sys/fs/cgroup cgroup.mode="Unified mode (Cgroupv2)"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Cgroupv2 supported controllers detected successfully" cgroup.controllers="[cpuset cpu io memory hugetlb pids rdma misc]" cgroup.fs=/sys/fs/cgroup cgroup.hierarchyID=0 cgroup.path=/proc/1/root/sys/fs/cgroup
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Cgroupv2 supported controllers detected successfully" cgroup.controllers="[cpuset cpu io memory pids]" cgroup.fs=/sys/fs/cgroup cgroup.hierarchyID=0 cgroup.path=/sys/fs/cgroup/system.slice/tetragon.service
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Cgroupv2 hierarchy validated successfully" cgroup.fs=/sys/fs/cgroup cgroup.path=/sys/fs/cgroup/system.slice/tetragon.service
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Deployment mode detection succeeded" cgroup.fs=/sys/fs/cgroup deployment.mode="systemd service"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Updated TetragonConf map successfully" NSPID=224406 cgroup.fs.magic=Cgroupv2 cgroup.hierarchyID=0 confmap-update=tg_conf_map deployment.mode="systemd service" log.level=info
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Disabling Kubernetes API"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Configured redaction filters" redactionFilters=
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Exit probe on acct_process"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Set execve_map entries 32768" size=27M
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="BTF file: using metadata file" metadata=/sys/kernel/btf/vmlinux
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Loading sensor" name=__base__
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Loading kernel version 6.8.12"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Loaded BPF maps and events for sensor successfully" maps="[Map{Name:execve_map PinPath:execve_map Owner:true} Map{Name:tg_execve_joined_info_map PinPath:tg_execve_joined_info_map Owner:true} Map{Name:execve_map_stats PinPath:execve_map_stats Owner:true} Map{Name:tg_execve_joined_info_map_stats PinPath:tg_execve_joined_info_map_stats Owner:true} Map{Name:execve_calls PinPath:__base__/event_execve/execve_calls Owner:true} Map{Name:tcpmon_map PinPath:tcpmon_map Owner:true} Map{Name:tg_conf_map PinPath:tg_conf_map Owner:true} Map{Name:tg_stats_map PinPath:tg_stats_map Owner:true} Map{Name:tg_mbset_map PinPath:tg_mbset_map Owner:true} Map{Name:tg_errmetrics_map PinPath:tg_errmetrics_map Owner:true}]" progs="[Program{Name:/usr/local/lib/tetragon/bpf/bpf_exit.o Attach:acct_process Label:kprobe/acct_process PinPath:__base__/event_exit} Program{Name:/usr/local/lib/tetragon/bpf/bpf_fork.o Attach:wake_up_new_task Label:kprobe/wake_up_new_task PinPath:__base__/kprobe_pid_clear} Program{Name:/usr/local/lib/tetragon/bpf/bpf_execve_event_v61.o Attach:sched/sched_process_exec Label:tracepoint/sys_execve PinPath:__base__/event_execve} Program{Name:/usr/local/lib/tetragon/bpf/bpf_execve_bprm_commit_creds.o Attach:security_bprm_committing_creds Label:kprobe/security_bprm_committing_creds PinPath:__base__/tg_kp_bprm_committing_creds}]" sensor=__base__
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Available sensors" sensors=__base__
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Registered sensors (policy-handlers)" policy-handlers="loader sensor, tracing, enforcer"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Registered probe types" types="execve, enforcer, generic_kprobe, generic_lsm, generic_tracepoint, generic_uprobe, loader"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Creating new EventCache" delay=2s retries=15
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Starting process manager" enableK8s=false enableProcessCred=false enableProcessNs=false
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Configured field filters" fieldFilters="[]"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Starting JSON exporter" logger="&{/var/log/tetragon/tetragon.log 10 0 5 false true -rw------- 0 <nil> {{} {0 0}} <nil> {{} {{} 0} {{} {0 0}}}}" request="allow_list:{event_set:PROCESS_KPROBE  event_set:PROCESS_UPROBE  event_set:PROCESS_TRACEPOINT  event_set:PROCESS_LSM}"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Exporter configuration" enabled=true fileName=/var/log/tetragon/tetragon.log
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Successfully detected bpftool path" bpftool=/usr/local/lib/tetragon/bpftool
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Successfully detected gops path" gops=/usr/local/lib/tetragon/gops
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="BPF: found active BPF resources" bpf-dir=/sys/fs/bpf/tetragon pinned-bpf="[__base__ execve_map execve_map_stats tcpmon_map tg_conf_map tg_errmetrics_map tg_execve_joined_info_map tg_execve_joined_info_map_stats tg_mbset_map tg_stats_map]"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Starting gRPC server" address=/var/run/tetragon/tetragon.sock protocol=unix
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Starting gRPC health server" address=":6789" interval=10
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Read ProcFS /proc/ appended 197/257 entries"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Maximum execve_map entries 32768, need to add 197."
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Cgroup rate disabled (0/0s)"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Unloading sensor __base__"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Sensor unloaded" maps="[Map{Name:execve_map PinPath:execve_map Owner:true} Map{Name:tg_execve_joined_info_map PinPath:tg_execve_joined_info_map Owner:true} Map{Name:execve_map_stats PinPath:execve_map_stats Owner:true} Map{Name:tg_execve_joined_info_map_stats PinPath:tg_execve_joined_info_map_stats Owner:true} Map{Name:execve_calls PinPath:__base__/event_execve/execve_calls Owner:true} Map{Name:tcpmon_map PinPath:tcpmon_map Owner:true} Map{Name:tg_conf_map PinPath:tg_conf_map Owner:true} Map{Name:tg_stats_map PinPath:tg_stats_map Owner:true} Map{Name:tg_mbset_map PinPath:tg_mbset_map Owner:true} Map{Name:tg_errmetrics_map PinPath:tg_errmetrics_map Owner:true}]" maps-error="[]" progs="[Program{Name:/usr/local/lib/tetragon/bpf/bpf_exit.o Attach:acct_process Label:kprobe/acct_process PinPath:__base__/event_exit} Program{Name:/usr/local/lib/tetragon/bpf/bpf_fork.o Attach:wake_up_new_task Label:kprobe/wake_up_new_task PinPath:__base__/kprobe_pid_clear} Program{Name:/usr/local/lib/tetragon/bpf/bpf_execve_event_v61.o Attach:sched/sched_process_exec Label:tracepoint/sys_execve PinPath:__base__/event_execve} Program{Name:/usr/local/lib/tetragon/bpf/bpf_execve_bprm_commit_creds.o Attach:security_bprm_committing_creds Label:kprobe/security_bprm_committing_creds PinPath:__base__/tg_kp_bprm_committing_creds}]"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="BPF events statistics: 0 received, 0% events loss"
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=info msg="Observer events statistics" errors=0 filterDrop=0 filterPass=0 lost=0 received=0
Jun 09 08:56:34 ip-10-55-0-100 tetragon[224406]: time="2025-06-09T08:56:34Z" level=fatal msg="Failed to start tetragon" error="policy handler 'tracing' failed loading policy 'create-below-dev-directory': validation failed: validateKprobeSpec: ksyms.KernelSymbols: no symbols found"
Jun 09 08:56:34 ip-10-55-0-100 systemd[1]: tetragon.service: Main process exited, code=exited, status=1/FAILURE
Jun 09 08:56:34 ip-10-55-0-100 systemd[1]: tetragon.service: Failed with result 'exit-code'.

Anything else?

Initial slack discussion https://cilium.slack.com/archives/C03EV7KJPJ9/p1749459691646069

quentinkhoo avatar Jun 10 '25 10:06 quentinkhoo

Thanks for reporting this, seems to be a change on how we treat the error from loading policies from a directory between v1.3 and v1.4 from the discussion on Slack. Let's see what changed, if it's intentional or not.

mtardy avatar Jun 10 '25 10:06 mtardy

so, i tried loading a policy via the tetragon binary with the --tracing-policy argument and unforunately, same error

time="2025-06-10T10:26:46Z" level=fatal msg="Failed to start tetragon" error="policy handler 'tracing' failed loading policy 'create-below-dev-directory': validation failed: validateKprobeSpec: ksyms.KernelSymbols: no symbols found"

in addition, i also tried to load it via tetra as shown below:

$ sudo ./tetra tp add /home/quentin.khoo/src/devops/ansible/roles/jump-host/templates/etc/tetragon/tetragon.tp.d/detect-create-below-dev-directory.yaml
Error: failed to add tracing policy: rpc error: code = Unknown desc = policy handler 'tracing' failed loading policy 'create-below-dev-directory': validation failed: validateKprobeSpec: ksyms.KernelSymbols: no symbols found

(admittedly, tetragon was running 1.4.0 but tetra was running 1.3.0. running tetra` 1.4.0 resulted in my own squid proxy error which i think is irrelevant for this issue, but i will still paste here anyway in case it's potentially related.)

$ sudo tetra tp add /home/quentin.khoo/src/devops/ansible/roles/jump-host/templates/etc/tetragon/tetragon.tp.d/detect-create-below-dev-directory.yaml
Error: failed to add tracing policy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: failed to do connect handshake, response: \"HTTP/1.1 400 Bad Request\\r\\nConnection: close\\r\\nContent-Length: 3336\\r\\nContent-Language: en\\r\\nContent-Type: text/html;charset=utf-8\\r\\nDate: Tue, 10 Jun 2025 10:38:50 GMT\\r\\nMime-Version: 1.0\\r\\nServer: squid/5.9\\r\\nVary: Accept-Language\\r\\nVia: 1.1 ip-10-55-0-100 (squid/5.9)\\r\\nX-Cache: MISS from ip-10-55-0-100\\r\\nX-Cache-Lookup: NONE from ip-10-55-0-100:3128\\r\\nX-Squid-Error: ERR_PROTOCOL_UNKNOWN 0\\r\\n\\r\\n<!DOCTYPE html PUBLIC \\\"-//W3C//DTD HTML 4.01//EN\\\" \\\"http://www.w3.org/TR/html4/strict.dtd\\\">\\n<html><head>\\n<meta type=\\\"copyright\\\" content=\\\"Copyright (C) 1996-2020 The Squid Software Foundation and contributors\\\">\\n<meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=utf-8\\\">\\n<title>ERROR: The requested URL could not be retrieved</title>\\n<style type=\\\"text/css\\\"><!-- \\n /*\\n * Copyright (C) 1996-2023 The Squid Software Foundation and contributors\\n *\\n * Squid software is distributed under GPLv2+ license and includes\\n * contributions from numerous individuals and organizations.\\n * Please see the COPYING and CONTRIBUTORS files for details.\\n */\\n\\n/*\\n Stylesheet for Squid Error pages\\n Adapted from design by Free CSS Templates\\n http://www.freecsstemplates.org\\n Released for free under a Creative Commons Attribution 2.5 License\\n*/\\n\\n/* Page basics */\\n* {\\n\\tfont-family: verdana, sans-serif;\\n}\\n\\nhtml body {\\n\\tmargin: 0;\\n\\tpadding: 0;\\n\\tbackground: #efefef;\\n\\tfont-size: 12px;\\n\\tcolor: #1e1e1e;\\n}\\n\\n/* Page displayed title area */\\n#titles {\\n\\tmargin-left: 15px;\\n\\tpadding: 10px;\\n\\tpadding-left: 100px;\\n\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\n}\\n\\n/* initial title */\\n#titles h1 {\\n\\tcolor: #000000;\\n}\\n#titles h2 {\\n\\tcolor: #000000;\\n}\\n\\n/* special event: FTP success page titles */\\n#titles ftpsuccess {\\n\\tbackground-color:#00ff00;\\n\\twidth:100%;\\n}\\n\\n/* Page displayed body content area */\\n#content {\\n\\tpadding: 10px;\\n\\tbackground: #ffffff;\\n}\\n\\n/* General text */\\np {\\n}\\n\\n/* error brief description */\\n#error p {\\n}\\n\\n/* some data which may have caused the problem */\\n#data {\\n}\\n\\n/* the error message received from the system or other software */\\n#sysmsg {\\n}\\n\\npre {\\n}\\n\\n/* special event: FTP / Gopher directory listing */\\n#dirmsg {\\n    font-family: courier, monospace;\\n    color: black;\\n    font-size: 10pt;\\n}\\n#dirlisting {\\n    margin-left: 2%;\\n    margin-right: 2%;\\n}\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\n    border-bottom: groove;\\n}\\n#dirlisting td.size {\\n    width: 50px;\\n    text-align: right;\\n    padding-right: 5px;\\n}\\n\\n/* horizontal lines */\\nhr {\\n\\tmargin: 0;\\n}\\n\\n/* page displayed footer area */\\n#footer {\\n\\tfont-size: 9px;\\n\\tpadding-left: 10px;\\n}\\n\\n\\nbody\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\n:lang(he) { direction: rtl; }\\n --></style>\\n</head><body id=ERR_PROTOCOL_UNKNOWN>\\n<div id=\\\"titles\\\">\\n<h1>ERROR</h1>\\n<h2>The requested URL could not be retrieved</h2>\\n</div>\\n<hr>\\n\\n<div id=\\\"content\\\">\\n<p>The following error was encountered while trying to retrieve the URL: <a href=\\\"error:invalid-request\\\">error:invalid-request</a></p>\\n\\n<blockquote id=\\\"error\\\">\\n<p><b>Unsupported Protocol</b></p>\\n</blockquote>\\n\\n<p>Squid does not support some access protocols. For example, the SSH protocol is currently not supported.</p>\\n\\n<p>Your cache administrator is <a href=\\\"mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_PROTOCOL_UNKNOWN&amp;body=CacheHost%3A%20ip-10-55-0-100%0D%0AErrPage%3A%20ERR_PROTOCOL_UNKNOWN%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Tue,%2010%20Jun%202025%2010%3A38%3A50%20GMT%0D%0A%0D%0AClientIP%3A%20127.0.0.1%0D%0A%0D%0AHTTP%20Request%3A%0D%0A%0D%0A%0D%0A\\\">webmaster</a>.</p>\\n<br>\\n</div>\\n\\n<hr>\\n<div id=\\\"footer\\\">\\n<p>Generated Tue, 10 Jun 2025 10:38:50 GMT by ip-10-55-0-100 (squid/5.9)</p>\\n<!-- ERR_PROTOCOL_UNKNOWN -->\\n</div>\\n</body></html>\\n\""

quentinkhoo avatar Jun 10 '25 10:06 quentinkhoo

Quentinkhoo provided feedback on Slack on this is issue:

hey guys, just wanted to share, i kinda figured out what was wrong and indeed it was a setting with my own system which i didn't consider :sweat_smile: as part of os_hardening i had kernel.kptr_restrict = 2 which disallowed reading to /proc/kallsyms properly so running sudo sysctl kernel.kptr_restrict = 0 solved everything :man-facepalming:

Here's a PR to close this so that's it easier in the future to figure out what went wrong https://github.com/cilium/tetragon/pull/3891.

mtardy avatar Jul 15 '25 14:07 mtardy