kubectl-trace
kubectl-trace copied to clipboard
Can't get trace on EKS with error "linux/types.h' file not found"
I have a vanilla EKS cluster and tried the first example here, that is:
$ kubectl trace run ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal \
-e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"
… however nothing happens. When I then look at the logs, I get:
$ kubectl trace logs kubectl-trace-ccf26276-03ef-11eb-bb6a-4865ee1b6baf if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
exit status 1
Any idea what I am doing wrong?
I also see this with GKE v1.18.6-gke.4801
nodeInfo:
architecture: amd64
bootID: 1d45c4af-5438-4d5e-a153-1036dc69a8cb
containerRuntimeVersion: docker://19.3.9
kernelVersion: 5.4.49+
kubeProxyVersion: v1.18.6-gke.4801
kubeletVersion: v1.18.6-gke.4801
machineID: ec7958c51f94158d4815fc14ad9f4614
operatingSystem: linux
osImage: Container-Optimized OS from Google
systemUUID: ec7958c5-1f94-158d-4815-fc14ad9f4614
Thanks for confirming @howardjohn and wondering if anyone here had a chance to look into it?
Is this project dead?
Can you give us the output of ls -la /lib/modules/$(uname -r)
please?
I think you need to install the kernel-devel-*
package on the node to get it working, BTW.
First attempt to try would be to use the --fetch-headers
flag.
Thanks @leodido for following up, clearly not dead ;)
Can you give us the output of ls -la /lib/modules/$(uname -r) please?
Will do ASAP, yes.
[ec2-user@ip-192-168-19-137 ~]$ ls -la /lib/modules/$(uname -r)
total 1200
drwxr-xr-x 7 root root 4096 Jul 24 16:02 .
drwxr-xr-x 3 root root 43 Jul 24 16:02 ..
lrwxrwxrwx 1 root root 46 Jul 24 16:02 build -> /usr/src/kernels/4.14.186-146.268.amzn2.x86_64
drwxr-xr-x 5 root root 42 Jul 24 16:02 extra
drwxr-xr-x 12 root root 130 Jul 24 16:02 kernel
-rw-r--r-- 1 root root 170617 Jul 24 16:02 modules.alias
-rw-r--r-- 1 root root 180101 Jul 24 16:02 modules.alias.bin
-rw-r--r-- 1 root root 292 Jul 14 18:19 modules.block
-rw-r--r-- 1 root root 3041 Jul 14 18:19 modules.builtin
-rw-r--r-- 1 root root 4927 Jul 24 16:02 modules.builtin.bin
-rw-r--r-- 1 root root 83649 Jul 24 16:02 modules.dep
-rw-r--r-- 1 root root 118851 Jul 24 16:02 modules.dep.bin
-rw-r--r-- 1 root root 383 Jul 24 16:02 modules.devname
-rw-r--r-- 1 root root 13 Jul 14 18:19 modules.drm
-rw-r--r-- 1 root root 13 Jul 14 18:19 modules.modesetting
-rw-r--r-- 1 root root 231 Jul 14 18:19 modules.networking
-rw-r--r-- 1 root root 32866 Jul 14 18:19 modules.order
-rw-r--r-- 1 root root 580 Jul 24 16:02 modules.softdep
-rw-r--r-- 1 root root 264809 Jul 24 16:02 modules.symbols
-rw-r--r-- 1 root root 322035 Jul 24 16:02 modules.symbols.bin
lrwxrwxrwx 1 root root 5 Jul 24 16:02 source -> build
drwxr-xr-x 2 root root 6 Jul 14 18:19 updates
drwxr-xr-x 3 root root 57 Jul 24 16:02 vdso
drwxr-xr-x 2 root root 6 Jul 14 18:19 weak-updates
As @leodido mentioned you may need to run with --fetch-headers
. I got the same issue on GKE until I started using --fetch-headers
.
GKE in M85 (1.18) should include BTF. I am going to be doing some work on bpftrace to ensure the image we use in kubectl trace can take advantage of this.
We also use GKE, which is why I made the --fetch-headers flag, since they don't have an easy way to get headers. However, once BTF is fully supported there, it will be much prefered.
So to get the tracing running properly you need to have linux-headers-$(uname -r)
installed on debian based systems; this correct? I assume that's the dependency for all nodes? Anything else need to be installed on the nodes? Just headers?
Headers yes, or use the “—fetch-headers” flag (which will slow down starting the trace significantly, but ought to ensure correctness).
support for BTF and IKHEADERS should be coming soon for kernels that support it.
for now, what we do at shopify is use a daemonset with an init container to pre-install headers on all nodes, using the same script as what is done with “—fetch-headers”.