Can't attach to kprobe/uprobe tracepoint

[olsajiri]

bpftrace won't attach to tracepoint event that provided for uprobe [1] or kprobe [2]

# perf probe -x ./test -a func 
Added new event:
  probe_test:func      (on func in /root/TTT/test)

You can now use it in all perf tools, such as:

        perf record -e probe_test:func -aR sleep 1 

# bpftrace -e 'tracepoint:probe_test:func { printf("%d\n", pid); }'
Attaching 1 probe...
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
ERROR: Error attaching probe: tracepoint:probe_test:func

Oleg suggested possible solution:

Perhaps bpftrace could look for probe_test:func in [uk]probe_events? Or simply retry ioctl(PERF_EVENT_IOC_SET_BPF) with BPF_PROG_TYPE_KPROBE if BPF_PROG_TYPE_TRACEPOINT returns EINVAL? Ugly, yes.

[1] https://lore.kernel.org/bpf/[email protected]/ [2] https://lore.kernel.org/bpf/Zt7Q6GVKtGTIdO1g@krava/

[ajor]

Issue also discussed in #3198

[mannkafai]

perf probe create [ku]probe_events in kernel via /sys/kernel/tracing/[ku]probe_events^1 . Libbpf crate perf_events with the type of PERF_TYPE_TRACEPOINT in userland to use this kind of [ku]probe_events, it's same whit tracepoints.

bpftrace -e 'tracepoint:probe_test:func { printf("%d\n", pid); }'

create a BPF program whit type of BPF_PROG_TYPE_TRACEPOINT, it’s right and works if probe_test:func is a tracepoint. But probe_test:func is not a tracepoint, it's an uprobe, the program type should be BPF_PROG_TYPE_KPROBE, so got an Invalid argument error when attach bpf program whit ioctl(PERF_EVENT_IOC_SET_BPF).

I think it's possible to add functions like [ku]probe:subsys:event to crate BPF_PROG_TYPE_KPROBE bpf program and attach as tracepoint like the legacy mode in libbpf.

@ajor

[viktormalik]

To sum it up (b/c I was a bit confused for a time while reading the above discussions): perf probe allows users to define custom "events" on top of existing uprobes and kprobes. These are registered via tracefs in /sys/kernel/tracing/[ku]probe_events.

Now the question is how to support these from bpftrace side. Based on the above discussion(s), I can see three options:

1. Support these events via the tracepoint provider as they are sort of "tracepoint events". That would mean implementing what Oleg suggests - tracepoint would not just look into /sys/kernel/tracing/events/* but also into sys/kernel/tracing/[ku]probe_events. This will come with several challenges as [ku]probe_events don't have format specification, arguments (i.e. args would be inaccessible in such tracepoints), and probably more. Basically, tracepoint would cover two kinds of events with quite a different semantics which is IMO not very nice.
2. Support these events via kprobe and uprobe providers as @mannkafai suggests. This makes the events semantics tied to the event type they are created on which is probably a good thing. The problem is that we'd probably have to expand kprobe and uprobe providers to support [ku]probe:subsys:event but the second part of the attach point is already used for different purpose (kernel module for kprobes, binary/library for uprobes).
3. Introduce a new probe type for these events, e.g. perf: , perf_probe:, probe:, or other.

So far option 2 seems the best to me but we'd have to solve the problem with attach points format.

[mannkafai]

This will come with several challenges as [ku]probe_events don't have format specification, arguments

/sys/kernel/tracing/[ku]probe_events shows the basic info (i.e. name, probe point) of [ku]probe_events which created via itself. Each of those [ku]probe_events has the same directory structure and files whit real tracepoints, kernel auto create dirs like probe_test/func/ under /sys/kernel/tracing/events/, so there is a format file in /sys/kernel/tracing/events/probe_test/func/ which describes format specification.

tracepoint would cover two kinds of events with quite a different semantics which is IMO not very nice.

I agree with you, tracepoints whit name of <subsys:event> which in /sys/kernel/tracing/[ku]probe_events are not real tracepoints. It could be two or three kinds of events whit option 1.

The problem is that we'd probably have to expand kprobe and uprobe providers to support [ku]probe:subsys:event but the second part of the attach point is already used for different purpose (kernel module for kprobes, binary/library for uprobes).

Yes, that's problem. I searched the code recently, the exist variants for [ku]probe can fit subsys:event , but attach point is different so it needs to be distinguished from the normal ones for creating probe, attaching BPF program and others, that may change a bit lot in current kprobe and uprobe providers IMO.

I think option 3 is the best just IMO.[ku]probe:subsys:event to kprobe is more like usdt to uprobe, it's for different purpose. Introduce new provider(s) can use the feature of tracepoint (i.e. format specification), could be easier and less complexity.

[danobi]

Just so I can understand the problem better, why are users creating perf probe events and then attaching to them with bpftrace? Seems like they could also either do all their tracing with perf or do all their tracing with bpftrace. Is there some use case for mixing the two?

[mannkafai]

Indeed, it is an uncommon use case in #3198, reuse the [ku]probe_events already set.

When I try perf probe in Ubuntu 24.04, comes out perf is not linked with libtraceevent, to use the new probe you can use tracefs:, perf can't trace the probe.