kubectl-trace

kubectl-trace copied to clipboard

I would like kubectl-trace to be able to use kernel headers from CONFIG_IKHEADERS. I am interested in supporting Minikube. I added CONFIG_IKHEADERS in Minikube (PR#8582; it is available in release v1.12.0-beta.1 but not in release v1.12.0, see regression in https://github.com/kubernetes/minikube/issues/8556#issuecomment-657205132).

If we get it to work, we can update the doc PR#8607 with examples from kubectl-trace in addition to the bcc & Inspektor Gadget examples.

bpftrace has PR#768 "utils: unpack kheaders.tar.xz if necessary" and it is included in kubectl-trace in commit https://github.com/iovisor/kubectl-trace/commit/2e97e6475a03d9b3340bbc12eddc8ee09cb76bca but it is unreleased yet.

Currently, kheaders.tar.xz cannot be read because xz is not installed:

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
tar (child): xz: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
/bpftrace/include/clang_workarounds.h:14:10: fatal error: 'linux/types.h' file not found
exit status 1

I tried to install it with the patch:

--- a/build/Dockerfile.tracerunner
+++ b/build/Dockerfile.tracerunner
@@ -13,6 +13,8 @@ RUN make _output/bin/trace-runner
 
 FROM ubuntu:19.10
 
+RUN apt-get install -y xz-utils
+
 COPY --from=gobuilder /go/src/github.com/iovisor/kubectl-trace/_output/bin/trace-runner /bin/trace-runner
 COPY --from=bpftrace /usr/bin/bpftrace /usr/bin/bpftrace
 

But it seems it still fail, none of the tracepoints work:

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
ERROR: tracepoint not found: syscalls:sys_enter_open
exit status 1

Tasks:

• [ ] Add CONFIG_IKHEADERS in Minikube again, see regression in https://github.com/kubernetes/minikube/issues/8556#issuecomment-657205132
• [ ] Install xz in the container image (https://github.com/iovisor/kubectl-trace/pull/123)
• [ ] Fix CONFIG_FTRACE_SYSCALLS in Minikube https://github.com/kubernetes/minikube/issues/8637
• [ ] Investigate next issues
• [ ] Make a new release of kubectl-trace
• [ ] Update Minikube documentation in PR#8607

cc @marga-kinvolk @mauriciovasquezbernal

Some events are available with Minikube but not syscalls:

$ ls -l /sys/kernel/debug/tracing/events/syscalls/
ls: cannot access '/sys/kernel/debug/tracing/events/syscalls/': No such file or directory
$ ls -l /sys/kernel/debug/tracing/events/|wc -l
94
$ zcat /proc/config.gz |grep -i CONFIG_FTRACE_SYSCALLS  
# CONFIG_FTRACE_SYSCALLS is not set

Strong +1