Make ebpf exporter to support CORE

[wenlxie]

This PR is for ebpf exporter to support CORE. This can make ebpf exporter to get CORE benefits and tolerate kernel changes.

I'd believe that there are more things need to do with this so please share your suggestions. Thanks.

[wenlxie]

@bobrik Can you help to take a look? thanks.

[bobrik]

There are lots of changes here and many of them are unrelated:

• Removed comments that still apply (starting with Dockerfile)
• Inconsistent indentation in C files
• Either missing or multiple newlines at the end of files
• Broken Go formatting, even on the lines you didn't really change
• Error strings starting with capital letters (only log lines should)
• Functions like tableValues moved around and generating large diff with no logical changes

Could you take care of this first to make it easier to look at the actual changes?

There are SPDX headers for LGPL, I'll need to ask legal people whether we can accept it.

I would also like to see some documentation in the README on how to use this.

[wenlxie]

@bobrik Thanks for the suggestion. I will take care of them. After move to CORE mode, BCC related will be removed, so I raised this PR for your to review first. Move from BCC to CORE is a right direction, but need your agreement with this.
Thanks again.

[wenlxie]

How to run ebpf_exporter with CORE mode?

1. Run make container generate the image directly, the default image is ebpf-exporter-build:lastest
2. Run the image ebpf-exporter-build:lastest with cmd like:

docker run -d --net=host --privileged -v /sys/kernel/debug:/sys/kernel/debug ebpf-exporter:latest

@bobrik

[wenlxie]

@bobrik I addressed some comments today but not finished all of these, will continue to do this tomorrow. Thanks . The image (https://github.com/cloudflare/ebpf_exporter/pull/130#issuecomment-1002443298 ) runs well on my workstation, you can have a try when you have time.

[rob-scheepens]

Due to not wanting to install Docker on all machines I intend to run ebpf_exporter on I tried to build on Bullseye vm as well. When running "./scripts/build-deb.sh release" I got:

dpkg-checkbuilddeps: error: Unmet build dependencies: python-netaddr

When trying to install python-netaddr:

root@rob-bullseye:~/bcc# apt-get install -y python-netaddr
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package python-netaddr is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  python3-netaddr

E: Package 'python-netaddr' has no installation candidate

Installing python3-netaddr works fine on Bullseye, but that doesn't address the dependency issue which comes from ./debian/control:

root@rob-bullseye:~/bcc# cat debian/control | grep netaddr
    python (>= 2.7), python-netaddr, python-pyroute2 | python3-pyroute2, luajit,

I tried changing to python3-netaddr, but still get the same error.

[bobrik]

@rob-scheepens, I think bcc build script checks out HEAD, so you need to commit your change to apply it.

Not sure how it's related to CO-RE.

[rob-scheepens]

@bobrik : Using the Dockerfile from https://github.com/wenlxie/ebpf_exporter/blob/upstream.master.core/Dockerfile I attempted to convert this to commands that can be scripted and run on a Debian Bullseye vm. All commands for now assume the user is root.

export PATH="/usr/lib/go-1.17/bin:$PATH"
export DEBIAN_FRONTEND=noninteractive
export LD_LIBRARY_PATH=/lib64

echo 'deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list

sed -i "s/bullseye main/bullseye main non-free/" /etc/apt/sources.list

apt-get update

apt-get --no-install-recommends install -y sudo devscripts build-essential fakeroot pbuilder aptitude git openssh-client ca-certificates git pkg-config libelf-dev libz-dev software-properties-common clang gnupg python3-netaddr pkg-config libz-dev software-properties-common bpftool python2 python3 dh-python libclang-9-dev libclang-common-9-dev libclang-cpp9 libclang1-9 libllvm9 llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools netperf python3-pyroute2 clang-format bison flex libfl-dev libedit-dev arping iperf ethtool python3 python3-doc python3-tk python3-venv python3.9-venv python3.9-doc cmake luajit libluajit-5.1-dev debhelper
apt-get install -y -t bullseye-backports golang-1.17 libbpfcc-dev libbpfcc bpfcc-tools

git clone --branch=v0.22.0 --depth=1 https://github.com/iovisor/bcc.git /root/bcc && \
git -C /root/bcc submodule update --init --recursive

git clone https://github.com/libbpf/libbpf.git /root/libbpf && \
cd /root/libbpf && git checkout 030ff87857090ae5c9d74859042d05bfb3b613a2 && cd src && mkdir build root && BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install

At this point a modification to the debian/control file is needed, see https://github.com/rob-scheepens/bcc/commit/6335995dec421e60298651cb6120d5bec247bd23 and my previous commit on that file, changing python2 dependencies as these do not work on Bullseye anymore. Script to make the changes and commit them (since the build-deb.sh script checks out HEAD):

cd /root/bcc/
git config --global user.email "user@[email.com](http://email.com/)"
git config --global [user.name](http://user.name/) "nutanix"
sed -i "s/python (>= 2.7)/python3/" debian/control
sed -i "s/python-netaddr/python3-netaddr/" debian/control
git add debian/control
git commit -m "python3 changes."
./scripts/build-deb.sh release

git clone https://github.com/wenlxie/ebpf_exporter.git /root/ebpf_exporter
cd /root/ebpf_exporter
git checkout upstream.master.core

This gets me up to the point where I run into the issue described in https://github.com/aquasecurity/libbpfgo/issues/1, where "go install" failed. Closer inspection learned that the -I include path in CFLAGS did not exist. Changing it to /root/libbpf/src/ made things progress more, up to the point where it fails on vmlinux.h missing:

    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
cd /root/ebpf_exporter/examples/CORE && make all
github.com/aquasecurity/libbpfgo
# github.com/aquasecurity/libbpfgo
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1153:37: could not determine kind of name for C.BPF_TC_CUSTOM
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1151:37: could not determine kind of name for C.BPF_TC_EGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1159:26: could not determine kind of name for C.BPF_TC_F_REPLACE
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1150:37: could not determine kind of name for C.BPF_TC_INGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1256:9: could not determine kind of name for C.bpf_tc_attach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1267:9: could not determine kind of name for C.bpf_tc_detach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1237:9: could not determine kind of name for C.bpf_tc_hook_create
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1246:9: could not determine kind of name for C.bpf_tc_hook_destroy
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1278:9: could not determine kind of name for C.bpf_tc_query
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1202:12: could not determine kind of name for C.sizeof_struct_bpf_tc_hook
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1179:12: could not determine kind of name for C.sizeof_struct_bpf_tc_opts
rm -f *.o
rm -rf config
clang -g -O2 -I"/tmp/libbpf-tools/.output" -I "/tmp/libbpf-tools/x86" -c   -target bpf -D__TARGET_ARCH_x86 biolatency.bpf.c -o biolatency.bpf.o
biolatency.bpf.c:2:10: fatal error: 'vmlinux.h' file not found
#include <vmlinux.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [Makefile:16: biolatency.bpf.o] Error 1

Using bpftool I generated vmlinux.h and put it in /tmp/libbpf-tools/ (is that correct?). Then I still got the C.BPF* errors as seen above:

root@rob-bullseye:~# cd /root/ebpf_exporter && CGO_CFLAGS="-I /root/libbpf/src/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
github.com/aquasecurity/libbpfgo
# github.com/aquasecurity/libbpfgo
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1153:37: could not determine kind of name for C.BPF_TC_CUSTOM
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1151:37: could not determine kind of name for C.BPF_TC_EGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1159:26: could not determine kind of name for C.BPF_TC_F_REPLACE
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1150:37: could not determine kind of name for C.BPF_TC_INGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1256:9: could not determine kind of name for C.bpf_tc_attach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1267:9: could not determine kind of name for C.bpf_tc_detach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1237:9: could not determine kind of name for C.bpf_tc_hook_create
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1246:9: could not determine kind of name for C.bpf_tc_hook_destroy
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1278:9: could not determine kind of name for C.bpf_tc_query
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1202:12: could not determine kind of name for C.sizeof_struct_bpf_tc_hook
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1179:12: could not determine kind of name for C.sizeof_struct_bpf_tc_opts

Digging further... updating libbpf-dev from bullseye-backports got rid of the above warnings, but still ebpf_exporter is not built:

root@rob-bullseye:~# cd /root/ebpf_exporter && CGO_CFLAGS="-I /root/libbpf/src/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
github.com/aquasecurity/libbpfgo
github.com/cloudflare/ebpf_exporter/exporter
github.com/cloudflare/ebpf_exporter/cmd/ebpf_exporter

Checking under vendor shows there is no cloudflare (yet):

root@rob-bullseye:~/ebpf_exporter/vendor/github.com# ls -l
total 32
drwxr-xr-x 4 root root 4096 Feb 16 08:54 alecthomas
drwxr-xr-x 3 root root 4096 Feb 16 08:54 aquasecurity
drwxr-xr-x 3 root root 4096 Feb 16 08:54 beorn7
drwxr-xr-x 3 root root 4096 Feb 16 08:54 cespare
drwxr-xr-x 3 root root 4096 Feb 16 08:54 golang
drwxr-xr-x 3 root root 4096 Feb 16 08:54 iovisor
drwxr-xr-x 3 root root 4096 Feb 16 08:54 matttproud
drwxr-xr-x 6 root root 4096 Feb 16 08:54 prometheus

Is that the reason the build is not complete?

Turns out ebpf_exporter binary is under /root/go/bin. :) Running it with config.yaml gives me this error:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/config.yaml
2022/02/16 10:07:10 Error attaching exporter: stat /root/ebpf_exporter/examples/CORE/biolatency.bpf.o: no such file or directory

The above error was resolved by running "make all" in examples/CORE. Then, when running, I got this error:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/config.yaml
libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf:
Unrecognized arg#0 type PTR
; hkey.numa_node = ctx->nid;
0: (61) r2 = *(u32 *)(r1 +8)
; hkey.numa_node = ctx->nid;
1: (63) *(u32 *)(r10 -8) = r2
; hkey.numa_zone = ctx->classzone_idx;
2: (85) call unknown#195896080
invalid func unknown#195896080
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

libbpf: -- END LOG --
libbpf: failed to load program 'tracepoint__compaction__mm_compaction_kcompactd_wake'
libbpf: failed to load object '/root/ebpf_exporter/examples/CORE/kcompactd.bpf.o'
2022/02/16 10:45:08 Error attaching exporter: Error to load program:kcompactd object  failed to load BPF object

This seems to be an issue with the config.yaml, when using biolatency.yaml it works:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/biolatency.yaml
2022/02/16 10:54:08 Starting with 1 programs found in the config
2022/02/16 10:54:08 Listening on :9435
^C

[rob-scheepens]

@bobrik : for now, this is the script I use to get it working on Bullseye:

export PATH="/usr/lib/go-1.17/bin:$PATH"
export DEBIAN_FRONTEND=noninteractive
export LD_LIBRARY_PATH=/lib64

# APT
echo 'deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list

sed -i "s/bullseye main/bullseye main non-free/" /etc/apt/sources.list

apt-get update && \

apt-get --no-install-recommends install -y sudo devscripts build-essential fakeroot pbuilder aptitude git openssh-client ca-certificates git pkg-config libelf-dev libz-dev software-properties-common clang gnupg python3-netaddr pkg-config libz-dev software-properties-common bpftool python2 python3 dh-python libclang-9-dev libclang-common-9-dev libclang-cpp9 libclang1-9 libllvm9 llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools netperf python3-pyroute2 clang-format bison flex libfl-dev libedit-dev arping iperf ethtool python3-doc python3-tk python3-venv python3.9-venv python3.9-doc cmake luajit libluajit-5.1-dev debhelper

apt-get install -y -t bullseye-backports golang-1.17 libbpfcc-dev libbpfcc bpfcc-tools libbpf-tools libbpf-dev

# BCC
git clone --branch=v0.22.0 --depth=1 https://github.com/iovisor/bcc.git /root/bcc && \
git -C /root/bcc submodule update --init --recursive

cd /root/bcc/
git config --global user.email "[email protected]"
git config --global user.name "nutanix"
sed -i "s/python (>= 2.7)/python3/" debian/control
sed -i "s/python-netaddr/python3-netaddr/" debian/control
git add debian/control
git commit -m "python3 changes."

cd /root/bcc && \
./scripts/build-deb.sh release

cp /root/bcc/libbcc_*.deb /tmp/libbcc.deb
cp -r /root/bcc/libbpf-tools  /tmp/libbpf-tools
dpkg -i /tmp/libbcc.deb

# libbpf
git clone --branch=v0.7.0 --depth=1 https://github.com/libbpf/libbpf.git && \
cd libbpf && cd src && mkdir build root && BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install

cp -r /root/libbpf/ /tmp/libbpf
cd /tmp/libbpf/src && make install

# ebpf_exporter
git clone https://github.com/wenlxie/ebpf_exporter.git /root/ebpf_exporter
cd /root/ebpf_exporter
git checkout upstream.master.core
cd /root/ebpf_exporter && CGO_CFLAGS="-I /usr/include/bpf/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
cd /root/ebpf_exporter/examples/CORE && make all
cp -r /root/ebpf_exporter/examples /root/examples
cp /root/go/bin/ebpf_exporter /usr/local/bin/ebpf_exporter
cd

# cleanup
find /tmp -name "*.git*"|xargs rm -Rf
find /root/examples -name "*.git*"|xargs rm -Rf

# RUN
ebpf_exporter --config.file=/root/examples/CORE/biolatency.yaml

cc: @wenlxie

[wenlxie]

I update the code:

1. Remove bcc dependency
2. Add tag support back
3. Make lib to be static linked
4. benchmark folder removed because of 1, will add it back

@bobrik @rob-scheepens Now ebpf_exporter compiled with lib static linked , it works well in my env.

root@4bfcb177e93f:~/go/bin# ldd ebpf_exporter
	not a dynamic executable

Appreciate that if you can help to have a try of this.

[rob-scheepens]

@wenlxie : using your latest upstream.master.core I checked again and the script above still works.

When you say "Remove bcc dependency", does that mean that there is no need to have bcc installed/built anymore, and that I can remove it from the script? When I tested that, things failed.

[wenlxie]

When you say "Remove bcc dependency", does that mean that there is no need to have bcc installed/built anymore, and that I can remove it from the script? When I tested that, things failed.

It means remove the bcc codes from the code/vendor

But it still need the header file in bcc to compile the bpf binary.

[rob-scheepens]

Unfortunately, I was not able to build this either with the Dockerfile or in a Debian Bullseye VM with libbpf from backports. Happy to have another look when I can build and run it myself.

@bobrik : can you give it another try using the script I provided for Bullseye?

cc: @wenlxie

[bobrik]

I was able to build this locally with some modifications. My plan is to clean this up and see how we can get this merged.

Apologies for the delay.

[wenlxie]

I was able to build this locally with some modifications. My plan is to clean this up and see how we can get this merged.

Apologies for the delay.

@bobrik Thanks for the test, please tell me with anything that I can help.

[bobrik]

https://github.com/aquasecurity/libbpfgo/pull/258 looks like it could be a good replacement for kaddr (or any argument) passing.

Not sure if we should wait for it though.

[bobrik]

Many thanks @wenlxie! I'll try to roll it out internally to iron any kinks before tagging a proper v2.0.0 release.

[bobrik]

It's out now: https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.0.0