pwru icon indicating copy to clipboard operation
pwru copied to clipboard

--filter-trace-tc option doesn't work with Cilium

Open hwchiu opened this issue 1 year ago • 6 comments

Hi, I’m running the pwru in the k8s server which running the cilium 1.11 version. everything works well but failed to start when I enabled the filter-trace-tc options, it showed the following error.

Failed to load objects: Verifier error: load program: invalid argument
func#0 @0
func#1 @37
func#2 @859
func#3 @1044
Subprog __send_drop_notify doesn't exist
Program fentry_tc: load program: invalid argument: Subprog __send_drop_notify doesn't exist (5 line(s) omitted

As I know, the _send_drop_notify is the function defined in the cilium code base and I don’t know how to fix it to ensure I can run the filter-trace-tc option with pwru in my server, any clue or options I can check?

Environment: OS: Ubuntu 20.04 Kernel: 5.15.0.69 pwru: master branch(2024/02/01).

hwchiu avatar Feb 02 '24 17:02 hwchiu

@hwchiu Thanks for the issue! Have you tried running with more recent Cilium version?

brb avatar Feb 14 '24 14:02 brb

@brb Thanks for the reply and unfortunately, I'm not able to upgrade the cilium version in that environment, but I will try to reproduce the issue in my home lab and then try the latest cilium to see if the issue still exists.

hwchiu avatar Feb 18 '24 12:02 hwchiu

Encountered the same issue on cilium 1.11.5, kernel 5.6

garyhuang123 avatar Feb 28 '24 08:02 garyhuang123

I just ran into this as well with an ebpf program I wrote. I compiled it with clang and embedded the btf information. Clearly the subprog in question exists.

What kind of extra info would be needed to investigate this? I can try to reduce my program to the minimal repro case.

wdullaer avatar May 17 '24 12:05 wdullaer

What kind of extra info would be needed to investigate this? I can try to reduce my program to the minimal repro case.

Which kernel and what Cilium vsn, what distro and how do you run K8s?

brb avatar Jun 11 '24 07:06 brb

The PR #383 fixes this issue.

This issue will happen when the very first function name in BTF info is out of the traced bpf prog.

I reproduce this case, and fix it by PR #383 .

sudo ./pwru --filter-trace-tc --filter-func '.*ip_out.*' --output-meta --output-tuple --output-limit-lines 10 icmp
2024/06/12 13:23:08 Failed to trace TC: failed to load objects: Verifier error: load program: invalid argument:
	func#0 @0
	func#1 @40
	func#2 @967
	func#3 @1024
	Subprog phy_on_ingress doesn't exist
	processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

program fentry_tc: load program: invalid argument: Subprog phy_on_ingress doesn't exist (5 line(s) omitted)

Asphaltt avatar Jun 12 '24 03:06 Asphaltt

The PR mentioned above has been merged.

brb avatar Jul 10 '24 15:07 brb