Build libbpf-bootstrap in `nixpkgs`

[nurdann]

I have changed Makefile to use already packaged libbpf and bpftool but I am running into a compilation issue inside nix-shell.

Building libbpf from submodule works, so I changed include (-I/nix/store/...-libbpf-.../include) and library (-l:libbpf:a paths

$ git log -1 --oneline
938651f (HEAD -> master, origin/master, origin/HEAD) rust: update Cargo dependencies
$ git diff
diff --git a/examples/c/Makefile b/examples/c/Makefile
index 548dd85..2ae6a55 100644
--- a/examples/c/Makefile
+++ b/examples/c/Makefile
@@ -4,9 +4,9 @@ CLANG ?= clang
 LLVM_STRIP ?= llvm-strip
 LIBBPF_SRC := $(abspath ../../libbpf/src)
 BPFTOOL_SRC := $(abspath ../../bpftool/src)
-LIBBPF_OBJ := $(abspath $(OUTPUT)/libbpf.a)
+LIBBPF_OBJ := /nix/store/sfx46bj2lamhqys9d80kbb4qck8qg6y1-libbpf-0.8.1/lib/libbpf.a
 BPFTOOL_OUTPUT ?= $(abspath $(OUTPUT)/bpftool)
-BPFTOOL ?= $(BPFTOOL_OUTPUT)/bootstrap/bpftool
+BPFTOOL ?= bpftool
 LIBBLAZESYM_SRC := $(abspath ../../blazesym/)
 LIBBLAZESYM_OBJ := $(abspath $(OUTPUT)/libblazesym.a)
 LIBBLAZESYM_HEADER := $(abspath $(OUTPUT)/blazesym.h)
@@ -15,7 +15,7 @@ VMLINUX := ../../vmlinux/$(ARCH)/vmlinux.h
 # Use our own libbpf API headers and Linux UAPI headers distributed with
 # libbpf to avoid dependency on system-wide headers, which could be missing or
 # outdated
-INCLUDES := -I$(OUTPUT) -I../../libbpf/include/uapi -I$(dir $(VMLINUX))
+INCLUDES := -I$(OUTPUT) -I/nix/store/sfx46bj2lamhqys9d80kbb4qck8qg6y1-libbpf-0.8.1/include -I$(dir $(VMLINUX))
 CFLAGS := -g -Wall
 ALL_LDFLAGS := $(LDFLAGS) $(EXTRA_LDFLAGS)
 
@@ -77,16 +77,10 @@ $(OUTPUT) $(OUTPUT)/libbpf $(BPFTOOL_OUTPUT):
 
 # Build libbpf
 $(LIBBPF_OBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(OUTPUT)/libbpf
-	$(call msg,LIB,$@)
-	$(Q)$(MAKE) -C $(LIBBPF_SRC) BUILD_STATIC_ONLY=1		      \
-		    OBJDIR=$(dir $@)/libbpf DESTDIR=$(dir $@)		      \
-		    INCLUDEDIR= LIBDIR= UAPIDIR=			      \
-		    install
-
+	:
 # Build bpftool
 $(BPFTOOL): | $(BPFTOOL_OUTPUT)
-	$(call msg,BPFTOOL,$@)
-	$(Q)$(MAKE) ARCH= CROSS_COMPILE= OUTPUT=$(BPFTOOL_OUTPUT)/ -C $(BPFTOOL_SRC) bootstrap
+	:
 
 
 $(LIBBLAZESYM_SRC)/target/release/libblazesym.a::

At the last stage when linking, I keep running into undefined reference to 'gelf_getnote'

$ nix-shell -p libelf zlib clang llvm libbpf bpftool

[nix-shell:~/libbpf-bootstrap/examples/c]$ make minimal
:
  BINARY   minimal
/nix/store/frriydwnf7v3dpayy5a6ki00qsp95iiy-binutils-2.38/bin/ld: /nix/store/sfx46bj2lamhqys9d80kbb4qck8qg6y1-libbpf-0.8.1/lib/libbpf.a(usdt.o): in function `usdt_manager_attach_usdt':
(.text+0xb7b): undefined reference to `gelf_getnote'
collect2: error: ld returned 1 exit status
make: *** [Makefile:122: minimal] Error 1

[chenhengqi]

Try install libelf-dev or libelf-devel.

[nurdann]

Try install libelf-dev or libelf-devel.

There is only libelf package available https://search.nixos.org/packages?channel=22.05&from=0&size=50&sort=relevance&type=packages&query=libelf

[chenhengqi]

Or elfutils ?

[nurdann]

Still the same error

$ nix-shell -p libbpf bpftool tree clang llvm libelf zlib elfutils
[nix-shell:~/libbpf-bootstrap/examples/c]$ make minimal
:
  BINARY   minimal
/nix/store/frriydwnf7v3dpayy5a6ki00qsp95iiy-binutils-2.38/bin/ld: /nix/store/sfx46bj2lamhqys9d80kbb4qck8qg6y1-libbpf-0.8.1/lib/libbpf.a(usdt.o): in function `usdt_manager_attach_usdt':
(.text+0xb7b): undefined reference to `gelf_getnote'
collect2: error: ld returned 1 exit status
make: *** [Makefile:122: minimal] Error 1

[0xB10C]

@nurdann for me, nix-shell --pure -p libbpf bpftool tree clang llvm elfutils zlib pkg-config (added pkg-config and dropped libelf) compiles (using submodules, not the nixpkgs libbpf and bpftools), but fails with libbpf: elf: minimal_bpf is not a valid eBPF object file as described in https://github.com/libbpf/libbpf-bootstrap/issues/105 on 22.05.

[0xB10C]

It might be that the packaged libbpf 0.8.1 on NixOS unstable your patch includes is to old? The submodules are v1.0 (see https://github.com/libbpf/libbpf-bootstrap/commit/e4c2098405ddb8a5f6d888ff95df933d321fc791). AFAIK there have been quite a few changes leading up to v1.0.

[anakryiko]

@nurdann for me, nix-shell --pure -p libbpf bpftool tree clang llvm elfutils zlib pkg-config (added pkg-config and dropped libelf) compiles (using submodules, not the nixpkgs libbpf and bpftools), but fails with libbpf: elf: minimal_bpf is not a valid eBPF object file as described in #105 on 22.05.

"minimal_bpf is not a valid eBPF object file" is reported if BPF object file is not ET_REL or it's e_machine isn't set to EM_BPF. I don't know all the reasons why this might happen, but very old Clang might be one of them.

[krisztianfekete]

FWIW, I get the same issue on my nixOS VM as well with these versions:

bpftool -V
bpftool v7.0.0
using libbpf v1.0
features: libbfd, libbpf_strict
(lorri)

clang -v
clang version 14.0.1
Target: aarch64-unknown-linux-gnu
Thread model: posix

[anakryiko]

@0xB10C can you share BPF object file that causes this problem?

@krisztianfekete do you run into "is not a valid eBPF object file" issue or the one with gelf_getnote? The latter means that whatever libelf you have on your system doesn't have expected API (which probably means your version is too old).

[krisztianfekete]

The former. This is my exact output:

[fktkrt@virtan1x:~/projects/bpf]$ bpftool version
bpftool v7.0.0
using libbpf v1.0
features: libbfd, libbpf_strict
(lorri)
[fktkrt@virtan1x:~/projects/bpf]$ clang -v
clang version 14.0.1
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/0865xqh42bdrgsqz3pd470fqjgswh6h7-clang-14.0.1/bin
(lorri)
[fktkrt@virtan1x:~/projects/bpf]$ cd libbpf-bootstrap/examples/c/
(lorri)
[fktkrt@virtan1x:~/projects/bpf/libbpf-bootstrap/examples/c]$ make
which: no cargo in (/nix/store/sw2sdbaz0i3zn6k0y4g54g42z2qhz3wv-patchelf-0.14.5/bin:/nix/store/rsziqsq7b2gmrzc9v1mmrcwla3qnh87q-gcc-wrapper-9.3.0/bin:/nix/store/z835nhs9j5k3nf4sdjc4d6d2ddcp6dns-gcc-9.3.0/bin:/nix/store/iwqwj8r6m3xdk6q0s6mj0dqr6qlypakw-glibc-2.34-210-bin/bin:/nix/store/hcww5hh0i9pjahb36kyia8iv23xvvz57-coreutils-9.0/bin:/nix/store/45byxgvwq2flycz5997s3q5lfzwm84ql-binutils-wrapper-2.38/bin:/nix/store/ync2dbd57gkchci8yqhl6p4aaz4ims1p-binutils-2.38/bin:/nix/store/4f8wbzggmmqyi026japim7p96g8nq6g1-hello-2.12/bin:/nix/store/zgyh7dj406sa07nz1pfz16gcs5i1zw4a-clang-wrapper-14.0.1/bin:/nix/store/0865xqh42bdrgsqz3pd470fqjgswh6h7-clang-14.0.1/bin:/nix/store/431k6lkp2dsxq31qyfjqva665s5kdi9d-binutils-wrapper-2.38/bin:/nix/store/wn9rpy833v6vd2pj28im87s8xav75n5y-llvm-11.1.0-dev/bin:/nix/store/2m92mhdyw1aqp5iila5s8as2c65ncnf9-ncurses-6.3-p20220507-dev/bin:/nix/store/hxb617pvn6h82j39bpza9vscr52g48p0-ncurses-6.3-p20220507/bin:/nix/store/lywf4chdgssbxrvb8w1gqgipls9klsbw-llvm-11.1.0/bin:/nix/store/y3mg4ap5lc1hqc3lc5yyj45a1hwjw7w4-elfutils-0.187-bin/bin:/nix/store/mfb4pxq6sqqx16h49i61jb09gsvnrr5b-pkg-config-wrapper-0.29.2/bin:/nix/store/rpi62sxy0m6s7n35mgw27f3yj3zvyhnc-bpftools-5.19.3/bin:/nix/store/r05mi3bw72c60gc7h4366ypqj0c38xz4-bash-interactive-5.1-p16/bin:/nix/store/yf1n051pmsp7yirsh96wbsyzwrmwsswd-compiler-rt-libc-14.0.1/bin:/nix/store/hcww5hh0i9pjahb36kyia8iv23xvvz57-coreutils-9.0/bin:/nix/store/mxysbddn9dn68v1sx40rkbg08gbwrfgg-findutils-4.9.0/bin:/nix/store/7fw5nxln4rl25myysy27rlbrvvkx4ip9-diffutils-3.8/bin:/nix/store/q8g3zw2ii8njhkps8kfa003c2dydxa6r-gnused-4.8/bin:/nix/store/2myk56rwbjdanxbf5ah7r1a1fdjapw78-gnugrep-3.7/bin:/nix/store/3f5g5r298xx9pxbc0a7c7826vf0317dq-gawk-5.1.1/bin:/nix/store/mahvc221qzgbhqmvq4sph5qc090hbkw0-gnutar-1.34/bin:/nix/store/5dmk7z67ccxjz3yqbk6bp7vivyw0j9xq-gzip-1.12/bin:/nix/store/npy0ypzkq5xk0jnddi8y5x2s0kk3p785-bzip2-1.0.6.0.2-bin/bin:/nix/store/axlw18pvphsa7z98j9pwmllrwa56dszv-gnumake-4.3/bin:/nix/store/ch44v7520g3m0ll1aamp4fhq5jg365c6-bash-5.1-p16/bin:/nix/store/fisn37wbdk608inzaag5k4lfla4a89zm-patch-2.7.6/bin:/nix/store/4kl9lizwil286pggskssdlxy327xdd72-xz-5.2.5-bin/bin:/run/wrappers/bin:/home/fktkrt/.nix-profile/bin:/etc/profiles/per-user/fktkrt/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin)
:
  GEN-SKEL .output/minimal.skel.h
libbpf: elf: minimal_bpf is not a valid eBPF object file
Error: failed to open BPF object file: BPF object format invalid
make: *** [Makefile:106: .output/minimal.skel.h] Error 95
make: *** Deleting file '.output/minimal.skel.h'
(lorri)

[anakryiko]

@krisztianfekete then same request, can you share generated minimal_bpf object file?

[0xB10C]

As mentioned in https://github.com/libbpf/libbpf-bootstrap/issues/105#issuecomment-1236092476, I think it's an issue with clang in NixOS (see https://github.com/NixOS/nixpkgs/issues/176128). Running readelf -h usdt.bpf.o | grep Machine on my usdt.bpf.o also reports Advanced Micro Devices X86-64 instead of Linux BPF. I haven't had a chance to play around with the patch, it might be fixed in the upcoming NixOS release.

Attached: usdt.bpf.o.gz

[krisztianfekete]

minimal.bpf.o.gz

[anakryiko]

@krisztianfekete yeah, your file has "Machine: AArch64". So it seems like compiler is ignoring -target bpf argument. I'd expect compiler to fail if target is unrecognized, but maybe Clang doesn't do that and instead just silently switched to native architecture. Either way, not much I can help with here, sorry.

[chenhengqi]

@0xB10C @krisztianfekete What's the output of llc --version in your systems ?

[krisztianfekete]

[fktkrt@virtan1x:~/projects/bpf]$ llc --version
LLVM (http://llvm.org/):
  LLVM version 11.1.0
  Optimized build.
  Default target: aarch64-unknown-linux-gnu
  Host CPU: (unknown)

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore
(lorri)

[0xB10C]

Still the same error

$ nix-shell -p libbpf bpftool tree clang llvm libelf zlib elfutils
[nix-shell:~/libbpf-bootstrap/examples/c]$ make minimal
:
  BINARY   minimal
/nix/store/frriydwnf7v3dpayy5a6ki00qsp95iiy-binutils-2.38/bin/ld: /nix/store/sfx46bj2lamhqys9d80kbb4qck8qg6y1-libbpf-0.8.1/lib/libbpf.a(usdt.o): in function `usdt_manager_attach_usdt':
(.text+0xb7b): undefined reference to `gelf_getnote'
collect2: error: ld returned 1 exit status
make: *** [Makefile:122: minimal] Error 1

@nurdann Just to make sure: Did you end up trying to build with elfutils but without libelf in your nix-shell as mentioned in https://github.com/libbpf/libbpf-bootstrap/issues/113#issuecomment-1261528890?

[nurdann]

There is error when generating skeleton header now

$ nix-shell -p libbpf bpftool tree clang llvm zlib elfutils
$ make minimal
...
  GEN-SKEL .output/minimal.skel.h
libbpf: elf: minimal_bpf is not a valid eBPF object file
Error: failed to open BPF object file: BPF object format invalid
make: *** [Makefile:112: .output/minimal.skel.h] Error 95
make: *** Deleting file '.output/minimal.skel.h'

[0xB10C]

Yes, see https://github.com/libbpf/libbpf-bootstrap/issues/105#issuecomment-1283886839. It's fixed in NixOS unstable.