User space static tracepoints with CO-RE

[jvijtiuk]

Hello,

is it possible to use libbpf and CO-RE to trace user space programs that have USDT probes defined with Systemtap's sys/sdt.h?

I've managed to build the simple example application with USDT probes as described in #327. However, I'm not sure what steps are required to attach a BPF CO-RE program to the probe, mainly what the SEC value should be and if anything other than the BPF program is required.

I've got the following so far

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

SEC("uprobe/tmp/trace_user/tick/loop")
int probe_tick_loop_1(struct pt_regs *reg)
{
	bpf_printk("Hello world, from trace!\n");
	return 0;
}

char LICENSE[] SEC("license") = "GPL";

So far various combinations I've tried for the section value didn't work. The user space loader that I'm using is a basic program that uses skel.h to open, load and attach the program and then reads the trace pipe at /sys/kernel/debug/tracing/trace_pipe. I'm pretty sure that isn't the issue, as depending on the SEC value, the BPF program either gets loaded as a KPROBE type, UNSPEC type, or doesn't get loaded at all.

I've started looking at the trace.py and the corresponding BCC source, but I'm not sure where exactly I should look yet. Running the trace tool in verbose mode generates a program similar to the one I've provided above, and the following function with it:

./trace -v 'u:/tmp/trace_user:loop "%u", arg1' -T -p $(pidof trace_user)
...
probe.usdt.get_text() =  #include <uapi/linux/ptrace.h>
static __always_inline int _bpf_readarg_probe_loop_1_1(struct pt_regs *ctx, void *dest, size_t len) {
  if (len != sizeof(int32_t)) return -1;
  { u64 __addr = ctx->bp + -4; __asm__ __volatile__("": : :"memory"); int32_t __res = 0x0; bpf_probe_read(&__res, sizeof(__res), (void *)__addr); *((int32_t *)dest) = __res; }
  return 0;
}

I've previously found the place where that is generated in the BCC source, but couldn't find any documentation on what exactly it is supposed to do.

Thanks, Juraj

[anakryiko]

If you already parsed SDT definition and have program that can read input arguments correctly, attaching to USDT is quite simple. Use SEC("uprobe/whatever"), so that libbpf knows it's supposed to be uprobe/kprobe. Then from user-space, use bpf_program__attach_uprobe() and provide correct binary path and offset.

Making CO-RE and libbpf work with USDT is not trivial due to highly variable nature of USDT argument locations and static, compile-once nature of BPF CO-RE programs. But I actually have a working local prototype that does that, even if it's a very different way than what BCC does. I'll try to find time this year to complete it and make it available through libbpf. But for now you either have to do all of that on your own, or stick with BCC for now.

[0xB10C]

hey @anakryiko! did you have a chance to work on / complete your prototype? If yes, can you link it here? :slightly_smiling_face:

[anakryiko]

No, not yet. But now that BPF static linker is more or less done, I'll try to do USDT for CO-RE next.

[shiyiyuedeyu]

Exciting work！I am looking forward to try CO-RE and libbpf with USDT!

[anakryiko]

@shiyiyuedeyu, if you feel adventurous and would like to get your hands on the very early working version, feel free to take a look at https://github.com/anakryiko/linux/commit/9bf5091f6d77474736008e74b01aa503ff3be5c9

I ran into some fundamental limitations with shared libraries (it's impossible to know actual IP of an USDT in a shared library, which I need to do lookups in BPF program) and thinking how to deal with them. But feel free to use that code as a starting point for your own experiments.

[gfx]

FYI there's an example program to use BPF CO-RE + USDT in libbpf/libbpf-bootstrap:

• https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/usdt.bpf.c
• https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/usdt.c

[shiyiyuedeyu]

Excellent work! I can't wait to try it！