issues met when debugging an x86_64 emulation process at the arm64 instruction-level

[foxsen]

When we follow https://ffri.github.io/ProjectChampollion/appendix/ to debug at arm64 instruction-level debug, there are many strange issues, such as:

• If a watchpoint is set, rosetta runtime will fail with "rosetta error: failed to allocate vm space for aot". An example session follows:

lisa@jjl bt % lldb -- ./runner ./float
(lldb) target create "./runner"
Current executable set to '/Users/lisa/bt/runner' (arm64).
(lldb) settings set -- target.run-args  "./float"
(lldb) r
Process 9182 launched: '/Users/lisa/bt/runner' (arm64)
Process 9182 stopped
* thread #2, stop reason = exec
    frame #0: 0x00007ffdfffbc3ec runtime`_mh_execute_header + 17388
runtime`_mh_execute_header:
->  0x7ffdfffbc3ec <+17388>: mov    x19, sp
    0x7ffdfffbc3f0 <+17392>: and    sp, x19, #0xfffffffffffffff0
    0x7ffdfffbc3f4 <+17396>: mov    x29, sp
    0x7ffdfffbc3f8 <+17400>: ldr    x20, [x19, #0x20]
Target 0: (runtime) stopped.
(lldb) watchpoint set expression 0x7ffdfffbc3f0    ====> set a watchpoint that won't hit
Watchpoint created: Watchpoint 1: addr = 0x7ffdfffbc3f0 size = 8 state = enabled ...
    new value: -7998388550590730625
(lldb) c
Process 9182 resuming
rosetta error: failed to allocate vm space for aot   =========> will cause such failure
Process 9182 stopped
* thread #2, stop reason = signal SIGTRAP
    frame #0: 0x00007ffdfffd4d38 runtime`_mh_execute_header + 118072
runtime`_mh_execute_header:
->  0x7ffdfffd4d38 <+118072>: brk    #0x1
    0x7ffdfffd4d3c <+118076>: stp    x20, x19, [sp, #-0x20]!
    0x7ffdfffd4d40 <+118080>: stp    x29, x30, [sp, #0x10]
    0x7ffdfffd4d44 <+118084>: add    x29, sp, #0x10            ; =0x10 
Target 0: (runtime) stopped.
(lldb) 

• command might struck. e.g.

isa@jjl bt % lldb -- ./runner ./lazy  ===> lazy has a dead loop in main()
(lldb) target create "./runner"
Current executable set to '/Users/lisa/bt/runner' (arm64).
(lldb) settings set -- target.run-args  "./lazy"
(lldb) r
Process 9235 launched: '/Users/lisa/bt/runner' (arm64)
Process 9235 stopped
* thread #2, stop reason = exec
    frame #0: 0x00007ffdfffbc3ec runtime`_mh_execute_header + 17388
runtime`_mh_execute_header:
->  0x7ffdfffbc3ec <+17388>: mov    x19, sp
    0x7ffdfffbc3f0 <+17392>: and    sp, x19, #0xfffffffffffffff0
    0x7ffdfffbc3f4 <+17396>: mov    x29, sp
    0x7ffdfffbc3f8 <+17400>: ldr    x20, [x19, #0x20]
Target 0: (runtime) stopped.
(lldb) c
Process 9235 resuming
Process 9235 stopped              ====> type ctrl+c to stop in the main function of lazy
* thread #2, stop reason = signal SIGSTOP
    frame #0: 0x0000000100011018
->  0x100011018: ldur   w0, [x5, #-0x8]
    0x10001101c: cmp    w0, #0x0                  ; =0x0 
    0x100011020: b.eq   0x100011028
    0x100011024: b      0x100011018
Target 0: (runtime) stopped.
(lldb) si        ========> stuck here

and so on.

[kohnakagawa]

Sorry for my slow response.

I know that such strange behavior exists, but I don't know any workaround for watchpoint issue. Breakpoint sometime works fine when we directly modify the file and embed brk #0x1 instruction.

[saagarjha]

My understanding of this issue is that LLDB creates mappings in the target process when trying to evaluate an expression, which makes Rosetta upset (you'll see similar behavior if you use the expression command). I've had luck using the "lower-level" commands that directly read registers and take addresses.

[kohnakagawa]

@saagarjha Thank you for sharing!

I've had luck using the "lower-level" commands that directly read registers and take addresses.

Could you show me concrete commands for enabling watchpoints? If you share this information, I'll update the project documentation.

[saagarjha]

Oh, I was mostly commenting in general terms–basically, recommending you use commands like register read and such that don't evaluate expressions. Unfortunately the watchpoint command does, so I wasn't able to use it directly. There's a Python API, but:

$ lldb arch -- -x86_64 ./a.out
(lldb) target create "arch"
Current executable set to 'arch' (arm64e).
(lldb) settings set -- target.run-args  "-x86_64" "./a.out"
(lldb) r
Process 8085 launched: '/usr/bin/arch' (arm64e)
Process 8085 stopped
* thread #2, stop reason = exec
    frame #0: 0x00007ff7fffbba2c runtime`_mh_execute_header + 14892
runtime`_mh_execute_header:
->  0x7ff7fffbba2c <+14892>: mov    x19, sp
    0x7ff7fffbba30 <+14896>: and    sp, x19, #0xfffffffffffffff0
    0x7ff7fffbba34 <+14900>: mov    x29, sp
    0x7ff7fffbba38 <+14904>: ldr    x20, [x19, #0x20]
Target 0: (runtime) stopped.
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> import lldb
>>> error = lldb.SBError()
>>> lldb.target.WatchAddress(0x7ff7fffea2f8, 1, True, True, error)
<lldb.SBWatchpoint; proxy of <Swig Object of type 'lldb::SBWatchpoint *' at 0x1038f03c0> >
>>> ^D
now exiting InteractiveConsole...
(lldb) c
Process 8085 resuming
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lldb                     0x00000001008082f0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  lldb                     0x0000000100807858 llvm::sys::RunSignalHandlers() + 128
2  lldb                     0x0000000100808b8c SignalHandler(int) + 292
3  libsystem_platform.dylib 0x00000001c39b04e4 _sigtramp + 56
4  LLDB                     0x000000010f21e788 lldb_private::ThreadPlanBase::ShouldStop(lldb_private::Event*) + 72
5  LLDB                     0x000000010f213e30 lldb_private::Thread::ShouldStop(lldb_private::Event*) + 1272
6  LLDB                     0x000000010f21b974 lldb_private::ThreadList::ShouldStop(lldb_private::Event*) + 824
7  LLDB                     0x000000010f1cc910 lldb_private::Process::ShouldBroadcastEvent(lldb_private::Event*) + 456
8  LLDB                     0x000000010f1c9248 lldb_private::Process::HandlePrivateEvent(std::__1::shared_ptr<lldb_private::Event>&) + 292
9  LLDB                     0x000000010f1cd5bc lldb_private::Process::RunPrivateStateThread(bool) + 1296
10 LLDB                     0x000000010f1ccccc lldb_private::Process::PrivateStateThread(void*) + 28
11 LLDB                     0x000000010f121e1c lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) + 104
12 libsystem_pthread.dylib  0x00000001c3999240 _pthread_start + 148
13 libsystem_pthread.dylib  0x00000001c3994024 thread_start + 8
********************
Crash reproducer for lldb-1300.0.42
Swift version 5.5.2-dev

Reproducer written to '/var/folders/x3/wncmj8p52t732rrxyz6h424m0000gn/T/reproducer-44d129'

Before attaching the reproducer to a bug report:
 - Look at the directory to ensure you're willing to share its content.
 - Make sure the reproducer works by replaying the reproducer.

Replay the reproducer with the following command:
/Applications/Xcode-beta.app/Contents/Developer/usr/bin/lldb -replay /var/folders/x3/wncmj8p52t732rrxyz6h424m0000gn/T/reproducer-44d129
********************
Segmentation fault: 11

…it seems to broken with the version of LLDB that's in Xcode. One built from source seems to work, but you'll run into issues once Rosetta sets up its exception server I believe, the same as breakpoints.

[kohnakagawa]

@saagarjha

Thank you for your detailed explanation. It seems difficult to use watchpoint in Arm64 instruction level... If you find any workaround for this, please update this issue thread (I will also look for it.).

recommending you use commands like register read and such that don't evaluate expressions.

If you have other knowledge about Rosetta 2 analysis, please let me know. Pull-requests are always welcome.

[romanheros]

Sorry for my slow response.

I know that such strange behavior exists, but I don't know any workaround for watchpoint issue. Breakpoint sometime works fine when we directly modify the file and embed brk #0x1 instruction.

I have trouble with breakpoint. I use lldb set an hardware breakpoint on AOT file by 'breakpoint set -a 0x100011000 -H'. It can issue. However, it stuck when I continue give a 'si' or 'continue' command.

Besides I modify the AOT file and embed 'brk #0x1' instruction in the AOT file. I try this way, but I get a trap in rosetta runtime, and the brk never issued.

Thanks for your delighting.

[saagarjha]

I still haven't figured out how to get breakpoints to work, because the exception server messes with them. I'll ask around to see if anyone else has figured it out.