[BUG] running no-core crashes on runtime

[roikol]

Prerequisites

• [ ] This affects latest released version.
• [x] This affects current development tree (origin/HEAD).
• [ ] There isn't an issue describing the bug.

Select one OR another:

• [ ] I'm going to create a PR to solve this (assign to yourself).
• [x] Someone else should solve this.

Bug description

i'm attempting to run tracee without CORE on older kernel (i've tried 4.18 and 5.2). tracee crashes right after it's starts running because of a go PANIC.

This is the error I'm getting:

ubuntu@ubuntu-VirtualBox:/tmp/test/tracee$ sudo ./dist/tracee-ebpf 
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffc0 pc=0x7f667c07176b]

runtime stack:
runtime.throw({0x165be14?, 0xd6600000000?})
	/snap/go/9848/src/runtime/panic.go:992 +0x71
runtime.sigpanic()
	/snap/go/9848/src/runtime/signal_unix.go:802 +0x3a9

goroutine 1 [syscall]:
runtime.cgocall(0x1361760, 0xc0010cf348)
	/snap/go/9848/src/runtime/cgocall.go:157 +0x5c fp=0xc0010cf320 sp=0xc0010cf2e8 pc=0x8e45dc
github.com/aquasecurity/libbpfgo._Cfunc_bpf_object__load(0x7f6640000c40)
	_cgo_gotypes.go:928 +0x4c fp=0xc0010cf348 sp=0xc0010cf320 pc=0xac2b0c
github.com/aquasecurity/libbpfgo.(*Module).BPFLoadObject.func1(0x0?)
	/root/go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:476 +0x46 fp=0xc0010cf380 sp=0xc0010cf348 pc=0xac5be6
github.com/aquasecurity/libbpfgo.(*Module).BPFLoadObject(0xc00137caf0?)
	/root/go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:476 +0x19 fp=0xc0010cf3b8 sp=0xc0010cf380 pc=0xac5b59
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPF(0xc0000001e0)
	/tmp/test/tracee/pkg/ebpf/tracee.go:898 +0x209 fp=0xc0010cf4a0 sp=0xc0010cf3b8 pc=0x12e6dc9
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0xc0000001e0)
	/tmp/test/tracee/pkg/ebpf/tracee.go:366 +0x369 fp=0xc0010cf610 sp=0xc0010cf4a0 pc=0x12e3c49
main.main.func1(0xc000571e40)
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:272 +0x1545 fp=0xc0010cfce0 sp=0xc0010cf610 pc=0x135cd25
github.com/urfave/cli/v2.(*App).RunContext(0xc0004ceea0, {0x1dc8e30?, 0xc0000440a0}, {0xc00003c220, 0x1, 0x1})
	/root/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:322 +0x953 fp=0xc0010cfe70 sp=0xc0010cfce0 pc=0x1345433
github.com/urfave/cli/v2.(*App).Run(...)
	/root/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:224
main.main()
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:364 +0xc7e fp=0xc0010cff80 sp=0xc0010cfe70 pc=0x135e4be
runtime.main()
	/snap/go/9848/src/runtime/proc.go:250 +0x212 fp=0xc0010cffe0 sp=0xc0010cff80 pc=0x91beb2
runtime.goexit()
	/snap/go/9848/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0010cffe8 sp=0xc0010cffe0 pc=0x94a2e1

goroutine 10 [syscall]:
os/signal.signal_recv()
	/snap/go/9848/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	/snap/go/9848/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	/snap/go/9848/src/os/signal/signal.go:151 +0x2a

goroutine 11 [select]:
main.main.func1.3()
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:243 +0x71
created by main.main.func1
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:242 +0x141e

goroutine 12 [select]:
main.main.func1.4()
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:253 +0x16a
created by main.main.func1
	/tmp/test/tracee/cmd/tracee-ebpf/main.go:250 +0x14be

goroutine 13 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0013b2680)
	/root/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x73
created by google.golang.org/grpc.newCCBalancerWrapper
	/root/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22a

goroutine 20 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0013b29c0)
	/root/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x73
created by google.golang.org/grpc.newCCBalancerWrapper
	/root/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22a

goroutine 18 [IO wait]:
internal/poll.runtime_pollWait(0x7f6654989950, 0x72)
	/snap/go/9848/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000564e00?, 0xc0013da000?, 0x0)
	/snap/go/9848/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/9848/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000564e00, {0xc0013da000, 0x8000, 0x8000})
	/snap/go/9848/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000564e00, {0xc0013da000?, 0x8ee405?, 0x3?})
	/snap/go/9848/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000011ea8, {0xc0013da000?, 0x30?, 0x10401?})
	/snap/go/9848/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0013c84e0, {0xc0004b3460, 0x9, 0xc000063800?})
	/snap/go/9848/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x1dbccc0, 0xc0013c84e0}, {0xc0004b3460, 0x9, 0x9}, 0x9)
	/snap/go/9848/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/snap/go/9848/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0004b3460?, 0x9?, 0xc0013ce540?}, {0x1dbccc0?, 0xc0013c84e0?})
	/root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004b3420)
	/root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0000003c0)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x414
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x193f

goroutine 19 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00137c500, 0x1)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:408 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0013c85a0)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:535 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x1f91

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0x7f6654989860, 0x72)
	/snap/go/9848/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000565200?, 0xc001408000?, 0x0)
	/snap/go/9848/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/9848/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000565200, {0xc001408000, 0x8000, 0x8000})
	/snap/go/9848/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000565200, {0xc001408000?, 0x8ee405?, 0x3?})
	/snap/go/9848/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000011f28, {0xc001408000?, 0x30?, 0x10401?})
	/snap/go/9848/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0013c9da0, {0xc0004b3540, 0x9, 0xc000062800?})
	/snap/go/9848/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x1dbccc0, 0xc0013c9da0}, {0xc0004b3540, 0x9, 0x9}, 0x9)
	/snap/go/9848/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/snap/go/9848/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0004b3540?, 0x9?, 0xc0013cedb0?}, {0x1dbccc0?, 0xc0013c9da0?})
	/root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004b3500)
	/root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000000780)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x414
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x193f

goroutine 23 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00137ca50, 0x1)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:408 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0013c9e00)
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:535 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x1f91

seems like the error occurs in libbpfgo func BPFLoadObject().

Steps to reproduce

Steps to reproduce the issue:

1. step 1: git clone https://github.com/aquasecurity/tracee.git
2. step 2: cd tracee && git submodule update --init
3. step 3: sudo make clean tracee-ebpf install-bpf-nocore
4. step 4: sudo ./dist/tracee-ebpf

Context

Relevant information about my setup:

OSInfo: ID: ubuntu OSInfo: ID_LIKE: debian OSInfo: PRETTY_NAME: "Ubuntu 20.04.2 LTS" OSInfo: VERSION_ID: "20.04" OSInfo: VERSION_CODENAME: focal OSInfo: KERNEL_RELEASE: 5.2.21-050221-generic OSInfo: ARCH: x86_64 OSInfo: VERSION: "20.04.2 LTS (Focal Fossa)"

Tracee version (or commit id of your tree): main LLVM version: 12.0.0 Golang version: go version go1.18.3 linux/amd64

Additional Information (files, logs, etc)

same thing also happened on ubuntu with kernel version 4.18

[rafaeldtinoco]

Just for context in here. We are currently testing/supporting:

Upstream:

And:

Focal (Ubuntu 5.4) Focal HWE (Ubuntu 5.13) Jammy (Ubuntu 5.15) Fedora 34 (Fedora 5.11) Fedora 35 (Fedora 5.14) Fedora 36 (Fedora 5.17) RHEL8/Stream8 (Rhel 4.18) RHEL9/Stream9 (Rhel 5.14)

So, if we want to support 5.2 (CO-RE or, like in this case, non CO-RE) as well we should stick with latest released upstream stable branch and not an Ubuntu released kernel (both are already EOL so it is difficult to guarantee they don't suffer from bugs that would disallow us running tracee because of kernel itself).

[yanivagman]

@roikol I see that this bug happens on bpf load and we recently updated libbpfgo (and libbpf), can you please check if it can be reproduced with main?

[yanivagman]

Support for non-CORE dropped in v0.14.0