failed to compile BPF module

[TahirAhmed916]

I am trying to run bashreadline.py but i got following error test@ubuntu:/usr/share/bcc/tools$ sudo python bashreadline.py /virtual/main.c:9:1: error: unsupported map type: maps/perf_output BPF_PERF_OUTPUT(events); ^ /virtual/include/bcc/helpers.h:64:4: note: expanded from macro 'BPF_PERF_OUTPUT' };
^ /virtual/main.c:19:5: error: expected initialized handle for bpf_table events.perf_submit(ctx,&data,sizeof(data)); ^ 2 errors generated. Traceback (most recent call last): File "bashreadline.py", line 51, in b = BPF(text=bpf_text) File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 126, in init raise Exception("Failed to compile BPF module %s" % src_file) Exception: Failed to compile BPF module

I am able to run some of tools. But in most of the cases i am facing same kind of error

[drzaeus77]

Usually this error is due to running an older kernel which doesn't support the perf events map type. Can you share which kernel you are running, and perhaps try upgrading to the latest stable kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/, since your shell says you are running Ubuntu?

[TahirAhmed916]

yes i am using 4.2.0-27-generic.

[drzaeus77]

For this particular tool, <4.4 kernel won't work. Ubuntu 16.04 (Xenial) should come with that by default, once it releases.

[goldshtn]

I suggest this issue be closed because we seem to have found the root cause.

[samof76]

I guess not... I have installed the bcc-tools with 4.4.19-29.55.amzn1.x86_64 linux kernel. And I still get this error.

[chyld]

I'm running Ubuntu 16.04 in a Docker container on a Mac OSX host:

Linux f44c63691935 4.4.20-moby #1 SMP Thu Sep 15 12:10:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I installed the tools using the following commands:

echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools

I'm trying to run

sudo ./hello_world.py

and getting the following error:

chdir(/lib/modules/4.4.20-moby/build): No such file or directory
Traceback (most recent call last):
  File "./hello_world.py", line 11, in <module>
    BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 240, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

I think my problem is I don't have any kernel headers, since I'm running in a Docker container. I'm not sure I can use BCC in this scenario.

[oviliz]

Same "Failed to compile BPF module" problem on a Scaleway VPS with their Xenial image.

[pchaigno]

@chyld See the QUICKSTART.md guide for how to run bcc in a Docker container.

[eljefedelrodeodeljefe]

@pchaigno that doesn't necessary. Maybe on a local machine, but e.g. in Kubernetes this will still fail.

I think without really big efforts toward this, adoption is hard, since this will like be a blocker for any container system.

[drzaeus77]

@eljefedelrodeodeljefe Do you have expectations of root permissions in your kube cluster? Most of the tools developed so far have an expectation of privilege, in which case I would push for an admin container containing the debug tools for the rest of the node. For instance, I would want a container which has bcc, perf, strace, and so on, at least some of which would need to include non-common dependencies (e.g. kernel headers, specific perf build). If that container exists, there is no reason why it shouldn't be able to monitor the rest of the node.

[eljefedelrodeodeljefe]

@drzaeus77 in k8s afaik all containers are privileged, at least in the younger version. I am gonna settle on exactly what you describe. k8s calls it daemonsets, that runs that one privileged container on the N number of nodes. It's easier to make it run and maintain it definitely. Initially I would have wanted to run bcc from within the application to mitigate centralised metrics and host resolution.

Thanks.

[Akagi201]

On ubuntu, sudo apt-get install -y linux-headers-<your kernel version> fix it.

[cmurf]

I'm hitting this on Fedora 26, with a 4.11.3 kernel.

[root@f26h tools]# ./btrfsslower chdir(/lib/modules/4.11.3-300.fc26.x86_64/build): No such file or directory Traceback (most recent call last): File "./btrfsslower", line 322, in b = BPF(text=bpf_text) File "/usr/lib/python3.6/site-packages/bcc/init.py", line 284, in init raise Exception("Failed to compile BPF module %s" % src_file) Exception: Failed to compile BPF module

[root@f26h tools]# ls -l /lib/modules/4.11.3-300.fc26.x86_64/build lrwxrwxrwx. 1 root root 39 May 25 13:01 /lib/modules/4.11.3-300.fc26.x86_64/build -> /usr/src/kernels/4.11.3-300.fc26.x86_64

That path is empty. Is kernel source really supposed to be installed?

OK, so 'dnf install kernel-headers' fixes this, I think that should be some kind of dependency figured out by Fedora packaging so I'll file a bug.

[axot]

I'm trying to run bcc in GKE which using cos. Here is how I compiled kernel headers for cos.

COS_VERSION=10452.109.0
gsutil cp gs://cos-tools/$COS_VERSION/kernel-src.tar.gz .

cat << EOF >> chromeos/config/base.config
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [optional, for tc filters]
CONFIG_NET_CLS_BPF=m
# [optional, for tc actions]
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_BPF_JIT=y
# [optional, for kprobes]
CONFIG_BPF_EVENTS=y
EOF

make oldconfig
make-kpkg kernel_headers

And then install bcc like this,

dpkg -i linux-headers-4.14.22_4.14.22-10.00.Custom_amd64.deb
mkdir -p /lib/modules/4.14.22+/
ln -s /usr/src/linux-headers-4.14.22 /lib/modules/4.14.22+/build

#apt-key adv --keyserver-options http-proxy=http://${PROXY}/ --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
curl -sS "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xD4284CDD" | sed -n "/^-----BEGIN/,/^-----END/p" > /tmp/key.pgp
apt-key add /tmp/key.pgp
echo "deb https://repo.iovisor.org/apt/xenial xenial main" | tee /etc/apt/sources.list.d/iovisor.list
apt-get update
apt-get install -y bcc-tools libbcc-examples

I got same is maps/hash map type enabled in your kernel? error. And advice?

# /usr/share/bcc/tools/biolatency
/virtual/main.c:9:1: error: could not open bpf map: Operation not permitted
is maps/hash map type enabled in your kernel?
BPF_HASH(start, struct request *);
^
/virtual/include/bcc/helpers.h:132:48: note: expanded from macro 'BPF_HASH'
  BPF_HASHX(__VA_ARGS__, BPF_HASH4, BPF_HASH3, BPF_HASH2, BPF_HASH1)(__VA_ARGS__)
                                               ^
/virtual/main.c:10:1: error: could not open bpf map: Operation not permitted
is maps/histogram map type enabled in your kernel?
BPF_HISTOGRAM(dist);
^
/virtual/include/bcc/helpers.h:177:48: note: expanded from macro 'BPF_HISTOGRAM'
  BPF_HISTX(__VA_ARGS__, BPF_HIST3, BPF_HIST2, BPF_HIST1)(__VA_ARGS__)
                                               ^
/virtual/main.c:16:5: error: bpf_table start failed to open
    start.update(&req, &ts);
    ^
/virtual/main.c:26:11: error: bpf_table start failed to open
    tsp = start.lookup(&req);
          ^
4 errors generated.
Traceback (most recent call last):
  File "/usr/share/bcc/tools/biolatency", line 114, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 318, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text

[dtakis]

Same problem with new kernels like 5.3 Is there any official guideline where we could install the library and have it working?

[matinats]

Same issue in centOS with: # uname -r 5.0.8-1.el7.elrepo.x86_64

Logs:

/usr/share/bcc/tools/capable -U

modprobe: FATAL: Module kheaders not found. chdir(/lib/modules/5.0.8-1.el7.elrepo.x86_64/build): No such file or directory Traceback (most recent call last): File "/usr/share/bcc/tools/capable", line 178, in b = BPF(text=bpf_text) File "/usr/lib/python2.7/site-packages/bcc/init.py", line 325, in init raise Exception("Failed to compile BPF text") Exception: Failed to compile BPF text

sysctl -a | grep bpf

kernel.unprivileged_bpf_disabled = 0 net.core.bpf_jit_enable = 1 net.core.bpf_jit_harden = 1 net.core.bpf_jit_kallsyms = 0 net.core.bpf_jit_limit = 398458880

I have installed everything: yum install kernel-devel dwarves bcc-devel bcc-tools python-bcc bcc

[yonghong-song]

The bcc tries to find /lib/modules/$(uname -r)/ directory for kernel-devel header files. You might need additional steps, esp. for upgraded kernel.

[TyKeeling]

@dtakis I was having the same problem with a 5.4.0-42-generic bionic.

Building from source fixed for me. Note that the tools are installed to /usr/share/bcc/tools. #2119 might also be helpful.

[rewanthtammana]

I'm facing same issue with kernel, 5.4.0-1051-aws

[rewanthtammana]

@TyKeeling Working like a charm. Thanks for the help.

[gewesp]

Hi, I'm getting a same/similar error with kernel 4.15.0; Ubuntu 18.04 with standard packages .

Symptoms:

ubuntu:~$ sudo execsnoop-bpfcc 
chdir(/lib/modules/4.15.0-50-generic/build): No such file or directory
Traceback (most recent call last):
  File "/usr/sbin/execsnoop-bpfcc", line 131, in <module>
    b = BPF(text=bpf_text.replace("MAXARG", args.max_args))
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 301, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module 
ubuntu:~$ uname -a
Linux hostname 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu:~$ cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

[harelix]

@TyKeeling - building from source is the way to go, thanks.

[federic000]

@TyKeeling compiling from the source also worked for me, thx