mips: problem accessing string arguments

[guidosarducci]

(this takes over discussion/tracking from https://github.com/iovisor/ply/pull/80#issuecomment-1358728266)

I was able to package and build ply under OpenWrt, then began testing across various MIPS archs and ARM32 by following posted examples from both ply and bpftrace. Most of the basics work, but on MIPS I'm not able to access and print string args.

I see the string/arg issue on mips32be, mips32le and mips64le, which seems to discount an endian or word-size bug as I first suspected. None of these problems appear on ARM32 however. The relevant kernel code and some logs illustrating the problem are below.

Kernel Code: (https://elixir.bootlin.com/linux/v5.15.83/source/fs/open.c#L1195)

static long do_sys_openat2(int dfd, const char __user *filename,
			   struct open_how *how)
{
	struct open_flags op;
	int fd = build_open_flags(how, &op);
	struct filename *tmp;

ARMVIRT32: (good)

root@OpenWrt:~# uname -a
Linux OpenWrt 5.15.83 #0 SMP Fri Dec 16 10:01:38 2022 armv7l GNU/Linux

root@OpenWrt:~# ply -T
Verifying kernel config (/proc/config.gz)... OK
  CONFIG_UPROBES is not set
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK
Verifying special... OK
Verifying interval... OK

root@OpenWrt:~# ply -k 'kprobe:do_sys_openat2 { printf("%v(%v): fd %v fn %v str %s\n", comm, uid, arg0, arg1, str(arg1)); }'
procd          (   0): fd 0xffffff9c fn 0x4eff1c str /dev
procd          (   0): fd 0x9 fn 0xb6e6b670 str ttyS0
procd          (   0): fd 0xffffff9c fn 0x4eff1c str /dev
procd          (   0): fd 0x9 fn 0xb6e6b670 str ttyS0
procd          (   0): fd 0xffffff9c fn 0x4eff1c str /dev
procd          (   0): fd 0x9 fn 0xb6e6b670 str ttyS0
askfirst       (   0): fd 0xffffff9c fn 0xbea648e0 str /etc/ld-musl-armhf.path
askfirst       (   0): fd 0xffffff9c fn 0xbea64a80 str /lib/libgcc_s.so.1

MIPS32BE: (bad)

root@OpenWrt:~# uname -a
Linux OpenWrt 5.15.83 #0 SMP Fri Dec 16 10:01:38 2022 mips GNU/Linux

root@OpenWrt:~# ply -T
Verifying kernel config (/proc/config.gz)... OK
  CONFIG_UPROBES is not set
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK
Verifying special... OK
Verifying interval... OK

root@OpenWrt:~# ply -k 'kprobe:do_sys_openat2 { printf("%v(%v): fd %v fn %v str %s\n", comm, uid, arg0, arg1, str(arg1)); }'
info: creating kallsyms cache
utpl           (   0): fd 0xffffff9c fn 0x77d54bf0 str
utpl           (   0): fd 0xffffff9c fn 0x77d43960 str
utpl           (   0): fd 0xffffff9c fn 0x7fcbc868 str
utpl           (   0): fd 0xffffff9c fn 0x7fcbc868 str
utpl           (   0): fd 0xffffff9c fn 0x77d54bf0 str
utpl           (   0): fd 0xffffff9c fn 0x7fcbc868 str
utpl           (   0): fd 0xffffff9c fn 0x7fcbc868 str

Could you please have a think about these symptoms? I suspect they may ring a bell given your familiarity with internals. I'd also appreciate if you could suggest other useful test cases e.g. printing chars from arrays, dumping the the pt_regs struct, testing the core str() function, etc.

A further point I'll note is that I'm running BTF-enabled builds after contributing this to OpenWrt, and the various struct pt_regs look as expected:

MIPS32BE:

struct pt_regs {
        long unsigned int pad0[8];
        long unsigned int regs[32];
        long unsigned int cp0_status;
        long unsigned int hi;
        long unsigned int lo;
        long unsigned int cp0_badvaddr;
        long unsigned int cp0_cause;
        long unsigned int cp0_epc;
        long unsigned int __last[0];
};

MIPS64LE:

struct pt_regs {
        long unsigned int regs[32];
        long unsigned int cp0_status;
        long unsigned int hi;
        long unsigned int lo;
        long unsigned int cp0_badvaddr;
        long unsigned int cp0_cause;
        long unsigned int cp0_epc;
        long unsigned int __last[0];
};

After further testing I find that things work fine on LTS kernel 5.10 for MIPS32BE, MIPS32LE and MIPS64LE. For example, MIPS32BE:

root@OpenWrt:/# uname -a
Linux OpenWrt 5.10.159 #0 SMP Wed Dec 21 01:39:50 2022 mips GNU/Linux

root@OpenWrt:/# ply -T
Verifying kernel config (/proc/config.gz)... OK
  CONFIG_UPROBES is not set
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK
Verifying special... OK
Verifying interval... OK

root@OpenWrt:/# ply -k 'kprobe:do_sys_openat2 { printf("%v(%v): fd %v fn %v str %s\n", comm, uid, arg0, arg1, str(arg1)); }'
uhttpd         (   0): fd 0xffffff9c fn 0x77bfce80 str /dev/urandom
uhttpd         (   0): fd 0xffffff9c fn 0x77bfce80 str /dev/urandom
uhttpd         (   0): fd 0xffffff9c fn 0x77bfce80 str /dev/urandom
luci           (   0): fd 0xffffff9c fn 0x7faeff08 str /etc/ld-musl-mips-sf.path
luci           (   0): fd 0xffffff9c fn 0x7faeff30 str /lib/libgcc_s.so.1
ucode          (   0): fd 0xffffff9c fn 0x7f9aaa78 str /etc/ld-musl-mips-sf.path
ucode          (   0): fd 0xffffff9c fn 0x7f9aaaa0 str /lib/libucode.so.20220812
ucode          (   0): fd 0xffffff9c fn 0x7f9aaaa0 str /usr/local/lib/libucode.so.20220812

It's possible something broke in the kernel between 5.10 and 5.15, or something was deliberately changed which ply does not accommodate. Looking next at upstream kernel and ply git histories...

[nikhilh-20]

I tried reverting https://github.com/iovisor/ply/commit/17864b9818cceca09f31a346908afe1c718c10c5 and the filepath strings seems to be printed on x86_64 kernel v5.10.175. This is for the ply master branch

[nikhilh-20]

Also works for kernel v5.15.58. Wasn't working before.

[wkz]

Sorry for the long delay on this. Could you try two changes to your script:

1. Change str() to mem(). This will force a copy of the memory pointed to by arg1, even if there are NULs in it.
2. Change the corresponding format specifier from %s to %v so that a hexdump will be generated if there are non-zero bytes in the copied memory.