Build of usdt example fails: "libbpf: elf: usdt_bpf is not a valid eBPF object file"

[0xB10C]

For me, building the USDT example fails with libbpf: elf: usdt_bpf is not a valid eBPF object file using elfutils (version 0.187) see https://github.com/libbpf/libbpf-bootstrap/pull/85#issuecomment-1162591490. I'm not sure if elfutils is the problem here, though.

Building the other examples, e.g., the minimal one, fails too with the same error.

$ make usdt
...
  MKDIR    bpftool
  BPFTOOL  bpftool/bootstrap/bpftool
...                        libbfd: [ OFF ]
...        disassembler-four-args: [ OFF ]
...                          zlib: [ on  ]
...                        libcap: [ OFF ]
...               clang-bpf-co-re: [ OFF ]
  CC       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/ringbuf.o
...
  CC       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/usdt.o
  AR       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/libbpf.a
  INSTALL  bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h skel_internal.h libbpf_version.h usdt.bpf.h
  CC       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/main.o
  CC       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/common.o
...
  CC       libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/disasm.o
  LINK     libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/bpftool
  GEN-SKEL .output/usdt.skel.h
libbpf: elf: usdt_bpf is not a valid eBPF object file
Error: failed to open BPF object file: BPF object format invalid
make: *** [Makefile:112: .output/usdt.skel.h] Error 95
make: *** Deleting file '.output/usdt.skel.h'

[chenhengqi]

Your clang is probably too old.

[0xB10C]

Thanks for helping to troubleshoot this. I'm using clang version 14.0.1 which is fairly recent. Do I need something newer?

[chenhengqi]

What's your distro and kernel version ?

It seams that clang-bpf-co-re: [ OFF ] causes the problem.

[0xB10C]

I'm on NixOS and kernel 5.15.55. I've just tested it in a Ubuntu 22.04 VM and it seems to work fine there. Though I'd like to get it running on NixOS too, if possible.

[0xB10C]

It seams that clang-bpf-co-re: [ OFF ] causes the problem.

Likely related to https://github.com/NixOS/nixpkgs/issues/176128. I'll revisit this and report back.

[welbonwang]

What's your distro and kernel version ?

It seams that clang-bpf-co-re: [ OFF ] causes the problem.

Chenheng, could you share your building environment? I met the same problem using Ubuntu 20.04.5 LTS with clang-10. Kernel is 5.15.0-48-generic with CONFIG_DEBUG_INFO_BTF=y

[chenhengqi]

Please upgrade your LLVM/clang.

[welbonwang]

I have built bpf grogram successfully using llvm-16 in teeks99/clang-ubuntu:16_20220910_1356 docker image.

[krisztianfekete]

Hey @0xB10C, have you managed to check this?

I switched to unstable to test the patch above myself, but seemingly it's still broken for me.

To reproduce, I have these:

Using unstable channel via niv:

[fktkrt@virtan1x:~/projects/bpf]$ cat nix/sources.json
{
    "nixpkgs": {
        "branch": "nixpkgs-unstable",
        "description": "Nix Packages collection",
        "homepage": "",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "27a89ba43b0fb735ce867e8ab3d2442f8cc61dad",
        "sha256": "0kmw4nhmmhx76hz82i1rnib06r9sik5zl4zsqh2ss9b1v9ydabnq",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/27a89ba43b0fb735ce867e8ab3d2442f8cc61dad.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    }
}
(lorri)

Check if niv is following unstable indeed:

[fktkrt@virtan1x:~/projects/bpf]$ niv show
nixpkgs
  branch: nixpkgs-unstable
  description: Nix Packages collection
  homepage:
  owner: NixOS
  repo: nixpkgs
  rev: 27a89ba43b0fb735ce867e8ab3d2442f8cc61dad
  sha256: 0kmw4nhmmhx76hz82i1rnib06r9sik5zl4zsqh2ss9b1v9ydabnq
  type: tarball
  url: https://github.com/NixOS/nixpkgs/archive/27a89ba43b0fb735ce867e8ab3d2442f8cc61dad.tar.gz
  url_template: https://github.com/<owner>/<repo>/archive/<rev>.tar.gz
(lorri)

My shell.nix:

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in

pkgs.mkShell {
  buildInputs = [
    pkgs.hello
    pkgs.clang
    pkgs.llvm
    pkgs.elfutils
    pkgs.zlib
    pkgs.pkg-config

    #pkgs.libbpf
    pkgs.bpftools

    # keep this line if you use bash
    pkgs.bashInteractive
  ];
}
(lorri)

And I still have this:

readelf -h .output/minimal.bpf.o | grep Machine
  Machine:                           AArch64

[0xB10C]

Thanks for testing with unstable. I haven't gotten around to work on this again. From what I understand, this isn't merged into master / unstable yet? It's merged into staging (https://github.com/NixOS/nixpkgs/issues/176128#issuecomment-1157132972), not sure how and when staging is merged into master in the NixOS release cycle.

[krisztianfekete]

My understanding is that this should be already available based on https://nixpk.gs/pr-tracker.html?pr=176152. But I might be wrong as well.

[0xB10C]

Good news! I'm able to build the examples with the clang_14 from NixOS unstable using the nix-shell below. I think https://github.com/NixOS/nixpkgs/issues/176128 indeed solved the issue. I had to disable the stackprotector hardening. I'm closing this issue, as it's solved for me.

let
  unstableTarball = fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz";
  pkgs = import <nixpkgs> {};
  unstable = import unstableTarball {};

  shell = pkgs.mkShell {

    # Building the examples doesn't work with the NixOS stack protector hardening (enabled by default):
    #   ...
    #   BPF      .output/minimal_legacy.bpf.o
    #   minimal_legacy.bpf.c:22:5: error: A call to built-in function '__stack_chk_fail' is not supported.
    # See stackprotector under https://nixos.org/manual/nixpkgs/stable/#sec-hardening-flags-enabled-by-default
    hardeningDisable = [ "stackprotector" ];

    buildInputs = [
        pkgs.tree
        unstable.clang_14
        pkgs.llvm
        pkgs.elfutils
        pkgs.zlib
        pkgs.pkg-config
        pkgs.which
    ];
  };
in shell