ci: uprobes integration test on Go binaries fails on kernel < 4.14

[mmat11]

This test should pass once this get fixed

[ti-mo]

@mmat11 Is there a plan to move forward with this? If you don't have the resources to investigate this in the near future, maybe best to add a comment to the API and close this up.

[mmat11]

Hi @ti-mo , I forgot about this issue (and haven't had much time to investigate anyway). I'll try to investigate it soon

[mmat11]

I've retested on a debian 9 VM and it works:

bpf prog
SEC("uprobe/traceme")
int uprobe_traceme(struct pt_regs *ctx) {
	bpf_printk("OK");
	return 0;
}

tracer
...
ex, err := link.OpenExecutable("/tmp/tracee")
if err != nil {
    log.Fatalf("opening executable: %s", err)
}

up, err := ex.Uretprobe("main.traceme", objs.UprobeTraceme, nil)
if err != nil {
    log.Fatalf("creating uprobe: %s", err)
}
defer up.Close()
...

tracee
func main() {
	for {
		time.Sleep(time.Second)
		traceme(1, 2)
	}
}

//go:noinline
func traceme(a, b int) {
	fmt.Println(a + b)
}

matt@debian9-univ:~$ /tmp/tracee
3
3
3
^C

matt@debian9-univ:~$ sudo cat /sys/kernel/debug/tracing/trace_pipe
[sudo] password for matt: 
tracee-7677  [006] ....  7341.527519: : OK
tracee-7677  [006] ....  7342.527641: : OK
tracee-7677  [006] ....  7343.527723: : OK

matt@debian9-univ:~$ uname -r
4.9.0-13-amd64

matt@debian9-univ:~$ cat /boot/config-4.9.0-13-amd64 |grep -e BPF -e UPROBE -e FUNCTION_TRACER
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_UPROBES=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_FUNCTION_TRACER=y
CONFIG_UPROBE_EVENT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m

So my guess would be that the CI kernel is missing some config?

[lmb]

From my most recent build:

lorenz@altair:~/dev/ci-kernels/build/linux-4.9.299 (master)
$ grep -e BPF -e UPROBE -e FUNCTION_TRACER .config
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_UPROBES=y
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_FUNCTION_TRACER=y
CONFIG_UPROBE_EVENT=y
CONFIG_BPF_EVENTS=y
# CONFIG_TEST_BPF is not set

[lmb]

Still doesn't work against the CI kernel:

$ ./run-tests.sh 4.9 -v -run UprobeProgramCall/go-binary ./link
Fetching linux-4.9.bz
Fetching linux-4.9-selftests-bpf.tgz
No selftests found, disabling
Testing on 4.9
qemu-system-x86_64: -fsdev local,id=virtfs1,path=/,security_model=none,readonly,multidevs=remap: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead
qemu-system-x86_64: -fsdev local,id=virtfs5,path=/home/lorenz/.local/lib/python3.9/site-packages/virtme/guest,security_model=none,readonly,multidevs=remap: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead
qemu-system-x86_64: -fsdev local,id=virtfs17,path=/tmp/tmp.fuZkDvjfpg,security_model=none,readonly,multidevs=remap: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead
qemu-system-x86_64: warning: 9p: degraded performance: a reasonable high msize should be chosen on client/guest side (chosen msize is <= 8192). See https://wiki.qemu.org/Documentation/9psetup#msize for details.
=== RUN   TestUprobeProgramCall
=== RUN   TestUprobeProgramCall/go-binary
    kprobe_test.go:365: unexpected value: want '1', got '0'
--- FAIL: TestUprobeProgramCall (0.04s)
    --- FAIL: TestUprobeProgramCall/go-binary (0.04s)
FAIL
FAIL	github.com/cilium/ebpf/link	1.110s
FAIL

[mmat11]

I've renamed the issue so it's less scary :). As written on slack, I've tested this manually with two different 4.9 kernels and the uprobe fires, so it's most likely some CI issue. I need some other time to understand what is wrong though

[lmb]

Yeah, fair enough. FWIW the uprobe readline example doesn't reliably trigger on my machine, maybe related?