x86_64 processes that execve i386 processes cause "Truncated register n in remote 'g' packet" on replay

[lf-]

I was debugging a system where a 32 bit executable is run by a wrapper script that runs in the host 64-bit bash, and I hit this issue. I minimized it to be the case of a 64 bit program exec'ing a 32 bit one in general.

hello.c:

#include <stdio.h>

int main(void) {
    printf("hello world\n");
}

test.c:

#include <stdio.h>
#include <unistd.h>

int main(void) {
    char *argv[] = {"hi", NULL};
    printf("test\n");
    execv("./hello", argv);
}

Compile:

gcc -m32 hello.c -o hello
gcc test.c

Reproduce:

/tmp » rr a.out
rr: Saving execution to trace directory `/home/jade/.local/share/
rr/a.out-5'.
test
hello world
/tmp » rr replay
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licens
es/gpl.html>
This is free software: you are free to change and redistribute it
.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/jade/.local/share/rr/a.out-5/mmap_copy
_5_a.out...
(No debugging symbols found in /home/jade/.local/share/rr/a.out-5
/mmap_copy_5_a.out)
Remote debugging using 127.0.0.1:61317
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)
BFD: warning: system-supplied DSO at 0x6fffd000 has a section ext
ending past end of file
0x00007f789f1ed090 in _start () from /lib64/ld-linux-x86-64.so.2
=> 0x00007f789f1ed090 <_start+0>:       48 89 e7        mov    rd
i,rsp
(rr) c
Continuing.
test
hello world
Truncated register 50 in remote 'g' packet
Truncated register 50 in remote 'g' packet
(rr) c
Continuing.
Truncated register 50 in remote 'g' packet
(rr)

Incidentally, if you want a bonus bug I accidentally found while looking at this one, change the initialization of argv in test.c to char *argv[] = NULL; and there will be fireworks when rr records the trace, but the program will execute normally outside rr, on my linux 5.13.10:

[FATAL /home/jade/.dotfiles/configs/pkg/rr-git/src/rr-git/src/record_syscall.cc:5984:rec_process_syscall_arch()] (task 3668707 (rec:3668707) at time 225) -> Assertion `t->regs().syscall_result_signed() == -syscall_state.expect_errno' failed to hold. Expected EFAULT for 'execve' but got result 0 (errno SUCCESS)

rr version: 1817b9d4

[bernhardu]

A workaround might be to just activate gdb after the process reached its "32-bit state" e.g. by "rr replay -g 200"

benutzer@debian:~/rr$ rr replay
...
0x00007fb617e49090 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) b execve
Function "execve" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (execve) pending.
(rr) cont
Continuing.
test

Breakpoint 1, execve () at ../sysdeps/unix/syscall-template.S:120
120     ../sysdeps/unix/syscall-template.S: Datei oder Verzeichnis nicht gefunden.
(rr) when
Current event: 181
(rr) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (process 630044) killed]
(rr) q

benutzer@debian:~/rr$ rr replay -g 183
test
...
Remote debugging using 127.0.0.1:40463

--------------------------------------------------
 ---> Reached target process 630044 at event 184.
--------------------------------------------------
Reading symbols from /lib/ld-linux.so.2...
Reading symbols from /usr/lib/debug/.build-id/84/18f45e8884ce73538211215a1cb607e0a635a6.debug...
0xf7f110b0 in _start () from /lib/ld-linux.so.2
(rr)

Another workaround might be to build a 32-bit rr with the cmake option: -Dforce32bit=true (But such a rr would not allow to record any 64-bit child process, so all recorded processes would need to be 32-bit.)

[bernhardu]

Just for completeness - gdbserver seems to be able to handle the transition from a 64 to a 32-bit process somehow:

benutzer@debian:~/rr$ gcc -O0 -g test.c
benutzer@debian:~/rr$ gcc -O0 -g -m32 hello.c -o hello
benutzer@debian:~/rr$ gdbserver localhost:1234 a.out
Process /home/benutzer/rr/a.out created; pid = 20214
Listening on port 1234
Remote debugging from host ::1, port 48230
test
benutzer@debian:~/rr$

benutzer@debian:~/rr$ gdb -q
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
Reading /home/benutzer/rr/a.out from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /home/benutzer/rr/a.out from remote target...
Reading symbols from target:/home/benutzer/rr/a.out...
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/32/438eb3b034da54caf58c7a65446639f7cfe274.debug...
0x00007ffff7fd3090 in _start () from target:/lib64/ld-linux-x86-64.so.2
(gdb) set breakpoint pending on
(gdb) set follow-fork-mode child
(gdb) set follow-exec-mode new
(gdb) b main
Breakpoint 1 at 0x55555555514d: file test.c, line 5.
(gdb) ignore 1 1
Will ignore next crossing of breakpoint 1.
(gdb) b execve
Function "execve" not defined.
Breakpoint 2 (execve) pending.
(gdb) cont
Continuing.
Reading /lib/x86_64-linux-gnu/libc.so.6 from remote target...

Breakpoint 2, execve () at ../sysdeps/unix/syscall-template.S:120
120     ../sysdeps/unix/syscall-template.S: Datei oder Verzeichnis nicht gefunden.
(gdb) pipe info target | grep "Local exec file" -A1
Local exec file:
        `target:/home/benutzer/rr/a.out', file type elf64-x86-64.
(gdb) display/i $pc
1: x/i $pc
=> 0x7ffff7ec96c0 <execve>:     mov    $0x3b,%eax
(gdb) stepi
0x00007ffff7ec96c5      120     in ../sysdeps/unix/syscall-template.S
1: x/i $pc
=> 0x7ffff7ec96c5 <execve+5>:   syscall 
(gdb) stepi
process 20214 is executing new program: /home/benutzer/rr/hello
[New inferior 2]
[New Thread 20214.20214]
Reading /home/benutzer/rr/hello from remote target...
Reading /home/benutzer/rr/hello from remote target...
Reading /lib/ld-linux.so.2 from remote target...
Reading /lib/ld-linux.so.2 from remote target...
Reading /lib/18f45e8884ce73538211215a1cb607e0a635a6.debug from remote target...
Reading /lib/.debug/18f45e8884ce73538211215a1cb607e0a635a6.debug from remote target...
Reading /usr/lib/debug//lib/18f45e8884ce73538211215a1cb607e0a635a6.debug from remote target...
Reading /usr/lib/debug/lib//18f45e8884ce73538211215a1cb607e0a635a6.debug from remote target...
Reading target:/usr/lib/debug/lib//18f45e8884ce73538211215a1cb607e0a635a6.debug from remote target...
Reading /lib32/libc.so.6 from remote target...
Reading /lib32/6320961f7f4181312f9149e33761b86fd3ce95.debug from remote target...
Reading /lib32/.debug/6320961f7f4181312f9149e33761b86fd3ce95.debug from remote target...
Reading /usr/lib/debug//lib32/6320961f7f4181312f9149e33761b86fd3ce95.debug from remote target...
Reading /usr/lib/debug/lib32//6320961f7f4181312f9149e33761b86fd3ce95.debug from remote target...
Reading target:/usr/lib/debug/lib32//6320961f7f4181312f9149e33761b86fd3ce95.debug from remote target...

Thread 2.1 "hello" hit Breakpoint 1, main () at hello.c:4
4           printf("hello world\n");
(gdb) pipe info target | grep "Local exec file" -A1
Local exec file:
        `target:/home/benutzer/rr/hello', file type elf32-i386.
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 2 (process 20214) killed]
(gdb) q
benutzer@debian:~/rr$

[rocallahan]

Generally we don't support continuing through an exec. So I don't get this behavior, I see us just stop at the exec without errors, which is expected.