How to compile latest ply version 2.1.1 with custom kernel-dir?

[ejparz]

Hi,

Ply looks like an awesome tool, and my embedded software team is very excited to try/use it.

We successfully cross-compiled the tool for arm-linux using arm-linux-androideabi-clang, and we pushed the tool to our embedded device.

The tool compiled correctly. i.e. the following two commands work on our arm embedded device:

./ply
error: no input
ply - Dynamic tracing utility

Usage:
  ply [options] <ply-text>
  ply [options] <ply-file>

Options:
  -c COMMAND     Run COMMAND in a shell, exit upon completion.
  -d             Enable debug output.
  -e             Exit after compiling.
  -h             Print usage message and exit.
  -k             Keep going in face of trace buffer overruns.
  -S             Show generated BPF.
  -v             Print version information.

./ply -v                                                   
ply 15c164b (linux-version:328450~5.3.2)

However, we are unable to run a basic ply test:

ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }' -d

When running the above command, we get the following error:

info: creating kallsyms cache
warning: unable to create kallsyms cache: No such file or directory
error: unable to load kprobe:do_sys_open, errno:22
warning: was ply built against the running kernel?
ERR:-22

In ply 1.x, there was a configure param --with-kerneldir that devs could use to set a custom kernel distro (other than kernel version on host build machine).

In 2.1.1 it seems this option is unrecognized as part of the configure process.

configure: WARNING: unrecognized options: --with-kerneldir

How can we compile ply 2.1.1 for the kernel version used on our embedded device, rather than the kernel version of our local development linux machines?

We are using kernel v4.14 on embedded device, but version 4.15 on our local dev ubuntu machines.

Thanks,

Eric

[ejparz]

In case it's helpful,

Here's part of an strace dump showing where the above command failed

strace -o strace_log.txt ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }' -d

write(2, "warning: unable to create kallsy"..., 42) = 42
write(2, "No such file or directory", 25) = 25
write(2, "\n", 1)                       = 1
ugetrlimit(RLIMIT_MEMLOCK, {rlim_cur=8192*1024, rlim_max=8192*1024}) = 0
setrlimit(RLIMIT_MEMLOCK, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
bpf(0, 0xbeff87c0, 0x70)                = 3
perf_event_open(0xbeff8788, -1, 0, -1, 0) = 4
bpf(0x2, 0xbeff8700, 0x70)              = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb35e1000
perf_event_open(0xbeff8788, -1, 1, -1, 0) = 5
bpf(0x2, 0xbeff8700, 0x70)              = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0xb35df000
perf_event_open(0xbeff8788, -1, 2, -1, 0) = 6
bpf(0x2, 0xbeff8700, 0x70)              = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0xb35d4000
perf_event_open(0xbeff8788, -1, 3, -1, 0) = 7
bpf(0x2, 0xbeff8700, 0x70)              = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0xb35d2000
mmap2(NULL, 16777216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb221b000
madvise(0xb221b000, 16777216, MADV_MERGEABLE) = 0
prctl(0x53564d41 /* PR_??? */, 0, 0xb221b000, 0x1000000, 0xb356681e) = 0
munmap(0xb221b000, 16777216)            = 0
mmap2(NULL, 17297408, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb219c000
madvise(0xb219c000, 17297408, MADV_MERGEABLE) = 0
prctl(0x53564d41 /* PR_??? */, 0, 0xb219c000, 0x107f000, 0xb356681e) = 0
munmap(0xb219c000, 409600)              = 0
munmap(0xb3200000, 110592)              = 0
bpf(0x5, 0xbeff87b0, 0x70)              = -1 EINVAL (Invalid argument)
write(2, "error: unable to load ", 22)  = 22
write(2, "kprobe:do_sys_open", 18)      = 18
write(2, ", errno:", 8)                 = 8
write(2, "22", 2)                       = 2
write(2, "\n", 1)                       = 1
write(2, "warning: was ply built against t"..., 51) = 51
madvise(0xb2200000, 16777216, MADV_DONTNEED) = 0
close(3)                                = 0
fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 0), ...}) = 0

Looks like the bpf syscall is not happy

bpf(0x5, 0xbeff87b0, 0x70)              = -1 EINVAL (Invalid argument)

[wkz]

The oldest kernel I can remember testing 2.x with is 4.19. It is not unlikely that ply is simply not compatible with older versions. Any chance you can test with a newer kernel?

[ydong08]

I hit the similar issue, my kernel version and ply version as follows:

# ply --version
ply 2.3.0-9-g1b57943-dirty (linux-version:328759~5.4.55)

ply dose work well with command:

# ply -d 'kprobe:fib_validate_source {print(stack);}'
debug: ir_bpf_generate: v0 -> r6
debug: unlimited memlock
debug: writing xprobe: p:ply7711/p18a9950_fib_validate_source fib_validate_source
debug: ply: active

        fib_validate_source
        ip_route_input_noref+156
        arp_process+964
        arp_rcv+396
        ...

however, ply reports error when i try other commands listed in README.md of ply project. debug info:

# ply -d 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
debug: ir_bpf_generate: v0 -> r6
debug: unlimited memlock
error: unable to load kprobe:do_sys_open, errno:22
error: output from kernel bpf verifier:
0: (bf) r6 = r1
1: (62) *(u32 *)(r10 -4) = 0
2: (7a) *(u64 *)(r10 -152) = 0
3: (7a) *(u64 *)(r10 -144) = 0
4: (bf) r1 = r10
5: (07) r1 += -152
6: (b7) r2 = 16
7: (85) call bpf_get_current_comm#16
last_idx 7 first_idx 0
regs=4 stack=0 before 6: (b7) r2 = 16
8: (85) call bpf_get_current_uid_gid#15
9: (57) r0 &= -1
10: (63) *(u32 *)(r10 -136) = r0
11: (bf) r3 = r6
12: (07) r3 += 8
13: (b7) r2 = 8
14: (bf) r1 = r10
15: (07) r1 += -168
16: (85) call unknown#113
invalid func unknown#113
processed 17 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1

ERR:-22

Any suggestion is appreciated.