Waiting indefinitely

[Sh4d1]

Hey!

When I try to run kubectl dig <my-node> it is waiting without suceeding (I've left it almost 1h waiting).

Any ideas on how to debug?

Cheers!

[fntlnz]

@Sh4d1 Oh thanks for bringing this up, we need to add some kind of status update for when it takes too long. It should be normally less than one second on with good bandwidth (the image is 700mb).

What dig does is that it creates a container, so you should see it with kubectl get pods and then you can get the logs for it as with anything else.

[Sh4d1]

Hmm, okay.

So I have: error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.

Guess I have to load the module :grin: maybe add it to the readme?

[fntlnz]

@Sh4d1 the module should load automatically, it probably had problems at downloading or compiling it in case it was not pre built.

I noticed the same problems too on machines that don’t have kernel headers for example, could you provide more info about the node like kernel version, Linux dietro and so on?

I think this is important to solve, thanks for reporting!

[Sh4d1]

Kernel: 4.15.0-50-generic Distro: Ubuntu 18.04.2 LTS (Bionic Beaver) I also don't have the kernel headers installed.

Also, maybe it's a new issue, but 9 times out of 10 when I try to log the pod (via docker on the node or via kubectl) it fucks up my terminal :disappointed:

[leodido]

I also don't have the kernel headers installed.

Can you please try installing headers?

[mjrlee]

I'm butting up against this on GKE too.

Also having the same problem where getting logs breaks my terminal (tmux/zsh/iTerm on Mac OS 10.14)

[zmarouf]

Same issue on Debian 8.10 kops AMI with the 4.4.121-k8s kernel and headers installed.

[fntlnz]

PR #4 solves this 🎉 can you please confirm @zmarouf @MartinLeedotOrg @Sh4d1 ?

kernel headers are still needed.

[leodido]

@zmarouf @MartinLeedotOrg @Sh4d1 have you had the opportunity to try it?

[Sh4d1]

@leodido not yet, I'll try to test it this week :)

[naiduarvind]

@leodido - tried it on an EKS cluster. Had to install the kernel header in the user data for the Amazon AMI and tried running kubectl dig however, it still did error out with the same message.

[leodido]

Thanks for feedback @naiduarvind.

This probably depends by the ko missing for the specific Amazon AMI. Having more details could help us investigating faster :)

[zmarouf]

I tried again on one of our test clusters and I'm still seeing the same error. AMI: k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11 (ami-4bfe6f33) Thank you for looking into this. It works well for other workloads though :)

[suside]

Same issue Ubuntu 16.04.3 LTS linux-headers-4.4.0-101-generic installed

$ kubectl logs -f kubectl-dig9fc21343-8dae-11e9-a5d9-cc2f716a3fd8-wfn8w 
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.

[mJace]

Stuck at a long waiting too, But my log says.

root@node1:~# kubectl logs kubectl-dig08476147-e101-11e9-b0c3-0cc47aeb8f5c-ccggf
can't open BPF probe '/root/.sysdig/sysdig-probe-bpf.o': No such file or directory

And seems like the driver is not able to load automatically.

root@node1:~# lsmod | grep sysdig
root@node1:~# 

[mJace]

I pre-installed the sysdig on my node1 And ok to run sysdig and csysdig on node1 by CLI But still unable to start kubectl-dig pod successfully.

the log of the error pod is still

can't open BPF probe '/root/.sysdig/sysdig-probe-bpf.o': No such file or directory