[BUG] apple m1 some events not working

[idanr1986]

Prerequisites

• [x] This affects latest released version.
• [x] This affects current development tree (origin/HEAD).
• [x] There isn't an issue describing the bug.

Select one OR another:

• [ ] I'm going to create a PR to solve this (assign to yourself).
• [x] Someone else should solve this.

Bug description

This is how I'm executing tracee (cmdline): ./tracee-ebpf -t e=hooked_proc_fops This is the error I'm getting: 2022/07/19 15:17:08 error creating Tracee: kernel symbols were not loaded currectly. Make sure tracee-ebpf has the CAP_SYSLOG capability

This is how I'm executing tracee (cmdline): ./tracee-ebpf -t e=hooked_syscalls This is the error I'm getting: 2022/07/19 15:17:08 error creating Tracee: kernel symbols were not loaded currectly. Make sure tracee-ebpf has the CAP_SYSLOG capability

This is how I'm executing tracee (cmdline): ./tracee-ebpf -t e=hooked_seq_ops This is the error I'm getting: 2022/07/19 15:17:08 error creating Tracee: kernel symbols were not loaded currectly. Make sure tracee-ebpf has the CAP_SYSLOG capability

This is how I'm executing tracee (cmdline): ./tracee-ebpf -t e=socket_dup This is the error I'm getting: 2022/07/14 06:21:29 error creating Tracee: failed to initialize tail call: failed to update map sys_exit_tails: operation not permitted

This is how I'm executing tracee (cmdline): ./tracee-ebpf -t e=kprobe_attach This is the error I'm getting: libbpf: kprobe perf_event_open() failed: No such file or directory libbpf: prog 'trace_arm_kprobe': failed to create kprobe 'arm_kprobe+0x0' perf event: No such file or directory 2022/07/14 06:21:13 error creating Tracee: failed to attach required probe: failed to attach event: arm_kprobe (failed to attach arm_kprobe k(ret)probe to program trace_arm_kprobe: no such file or directory)

Steps to reproduce the issue:

just run Tracee-ebpf with this events on linux arm

Context

Relevant information about my setup: *CPU:apple m1 - aarch64

• Linux version:Ubuntu 22.04 LTS
• Linux kernel version: 5.15.0-40-generic
• Tracee version (or commit id of your tree):main
• LLVM version:12
• Golang version:go version go1.18.3 linux/arm64

Additional Information (files, logs, etc)

[james-luther]

This bug also exists when deploying v0.8.0 to GKE. Pods aren't fully starting with the following error:

error creating Tracee: kernel symbols were not loaded currectly. Make sure tracee-ebpf has the CAP_SYSLOG capability

Slack link here: https://aquasecurity.slack.com/archives/C02LK3U786L/p1658937911322199

[rafaeldtinoco]

After fixing small compilation issue caused by recent changes, This is the current status for this issue:

hooked_proc_fops (OKAY, needs real test)

[ubuntu@ip-172-31-29-29 ~/tracee]$ sudo ./dist/tracee-ebpf -t e=hooked_proc_fops
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}

hooked_syscalls (BROKEN, will check)

[ubuntu@ip-172-31-29-29 ~/tracee]$ sudo ./dist/tracee-ebpf -t e=hooked_syscalls
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
failed to apply invoke context on print_syscall_table event: no event context with id 0

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:1 LostEvCount:0 LostWrCount:0 LostNtCount:0}

hooked_seq_ops (OKAY, needs real test)

[ubuntu@ip-172-31-29-29 ~/tracee]$ sudo ./dist/tracee-ebpf -t e=hooked_seq_ops
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}

socket_dup (Addressed by PR: #2103)

[ubuntu@ip-172-31-29-29 ~/tracee]$ sudo ./dist/tracee-ebpf -t e=socket_dup
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
2022/08/25 18:38:01 error initializing Tracee: failed to initialize tail call: failed to update map sys_exit_tails: operation not permitted

kprobe_attach (Addressed by PR: #2103)

[ubuntu@ip-172-31-29-29 ~/tracee]$ sudo ./dist/tracee-ebpf -t e=kprobe_attach
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
libbpf: kprobe perf_event_open() failed: No such file or directory
libbpf: prog 'trace_arm_kprobe': failed to create kprobe 'arm_kprobe+0x0' perf event: No such file or directory

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
2022/08/25 18:38:22 error initializing Tracee: failed to attach required probe: failed to attach event: arm_kprobe (failed to attach arm_kprobe k(ret)probe to program trace_arm_kprobe: no such file or directory)

[rafaeldtinoco]

The following error (in ARM64):

$ sudo ./dist/tracee-ebpf -t e=hooked_syscalls
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
failed to apply invoke context on print_syscall_table event: no event context with id 0

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:1 LostEvCount:0 LostWrCount:0 LostNtCount:0}

Happens because differences in golang calling convention among different golang versions (and architectures). The following code (and similar uprobe handlers):

SEC("uprobe/trigger_syscall_event")
int uprobe_syscall_trigger(struct pt_regs *ctx)
{
    u64 caller_ctx_id;
#if defined(bpf_target_x86)
    caller_ctx_id = ctx->bx;
#elif defined(bpf_target_arm64)
    bpf_probe_read(&caller_ctx_id, 8, ((void *) ctx->sp) + 16);
#else
    return 0;
#endif

Are wrong if we compile tracee with 1.18 and 1.19 golang version, when we would have to use:

caller_ctx_id = ctx->user_regs.regs[1];

to get arguments for the functions being uprobed.

[AsafEitani]

The following error (in ARM64):

$ sudo ./dist/tracee-ebpf -t e=hooked_syscalls
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
failed to apply invoke context on print_syscall_table event: no event context with id 0

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:1 LostEvCount:0 LostWrCount:0 LostNtCount:0}

Happens because the golang calling convention has a complex algorithm to decide where to pass arguments. @AsafEitani, you set the following:

SEC("uprobe/trigger_syscall_event")
int uprobe_syscall_trigger(struct pt_regs *ctx)
{
    u64 caller_ctx_id;
#if defined(bpf_target_x86)
    caller_ctx_id = ctx->bx;
#elif defined(bpf_target_arm64)
    bpf_probe_read(&caller_ctx_id, 8, ((void *) ctx->sp) + 16);
#else
    return 0;
#endif

But I can only get the correct context when picking the variable from registers:

caller_ctx_id = ctx->user_regs.regs[1];

for my ARM64 env, and not from the stack. Could you check and let me know ? This is part of fixes for the ARM64 that I'm preparing (A patchset).

That's weird, I assume golang differs the argument passing from different version\OS as well. I tested it on a ubuntu focal ARM and by this documentation - https://go.googlesource.com/go/+/refs/heads/dev.regabi/src/cmd/compile/internal-abi.md

As well as on my ubuntu focal x86 64 bit. Maybe it differs due to the M1 or it being a mac... Hard to tell..

[rafaeldtinoco]

Tracking the ARM64 uprobe argument issue at: https://github.com/aquasecurity/tracee/issues/2124