ndk
ndk copied to clipboard
rust-mobile
Android backtraces don't work since gimli is used as the symbolizer
Got this non-informative backtrace in logcat:
12-16 22:26:30.581 12603 12619 I RustStdoutStderr: thread '<unnamed>' panicked at 'called 'Result::unwrap()' on an 'Err' value: GlutinCreationError(NoAvailablePixelFormat)', src/main.rs:74:71
12-16 22:26:30.581 12603 12619 I RustStdoutStderr: stack backtrace:
12-16 22:26:30.581 12603 12619 I RustStdoutStderr: 0: 0xdd581fee - <unknown>
12-16 22:26:30.581 12603 12619 I RustStdoutStderr: 1: 0xdd5966c6 - <unknown>
.....
12-16 22:26:30.584 12603 12619 I RustStdoutStderr: 25: 0xe957757e - <unknown>
Then, while trying
cargo apk gdb
got this
Remote communication error. Target disconnected.: Connection reset by peer.
(gdb) quit
Found in log
12-16 22:22:10.125 31207 31207 E adbd : failed to connect to socket 'localfilesystem:/data/data/rust.conrod_android_skeleton/debug_socket': Connection refused
12-16 22:22:10.378 31207 31207 E adbd : failed to connect to socket 'localfilesystem:/data/data/rust.conrod_android_skeleton/debug_socket': Connection refused
The app conrod_android_skeleton was set as "Debugging app" in settings (also same result was when it wasn't set). So, it obviously is a problem on my side. But I'd like to have an option for attaching all debug symbols right into binary.
Hi, thanks for the report! Do you have a repository for testing it locally by chance?
https://github.com/inferrna/conrod-android-skeleton
it's my tries to actualise this https://github.com/jbg/conrod-android-skeleton
Hmm the missing backtrace symbolization seems to be a limitation of the backtrace library right now, but I will look a bit more around. Apparently, the rust compiler switched from libbacktrace to gimli this year and android support seems to be lacking right now: https://github.com/rust-lang/backtrace-rs/issues/351
apparently there is support in backtrace-rs now. according to https://github.com/rust-lang/backtrace-rs/issues/442 some changes to the AndroidManifest.xml are required. those should probably be added in debug builds. however I still can't get it to work
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
package="com.myapp">
<application
tools:replace="android:extractNativeLibs"
android:extractNativeLibs="true" />
</manifest>
Just to mention that I'm seeing the same issue and also tried setting android:extractNativeLibs="true" as was mentioned as a workaround under https://github.com/rust-lang/backtrace-rs/issues/442 but that didn't seem to help.
I've tried with ndk 22b and 23b so far and I'm building for arm64-v8a.
As far as I could tell the library had debug symbols (checked with nm and readelf). I also double checked symbols weren't being stripped by extracting the library from the apk and checked that file had symbols.
I'm not so sure now that backtrace-rs is really expected to support symbolizing a backtrace on Android (see https://github.com/rust-lang/backtrace-rs/issues/442#issuecomment-942396909) - I think it just tries to support getting the library offset for each stack frame.
Actually in my case I'm seeing that even the offsets aren't valid - they are far to large - even if I use the legacy apk packaging to ensure the shared libraries are separate files.
What I've realized I can do as a workaround though is to check for the backtrace that's automatically printed by Android itself because that at least has correct addresses / offsets. I then wrote a python script that just scrapes the offsets which can be piped to addr2line in the NDK:
#!/usr/bin/python3
import sys
import re
for line in sys.stdin:
search = re.search('#[0-9]+ +pc +([0-9A-Fa-f]+) +', line)
if search != None:
print(search.group(1))
So then I can copy paste a backtrace into a file and get a backtrace with filenames, demangled functions and line numbers via:
cat backtrace.txt | py ./android-backtrace.py | 'path/to/toolchains/llvm/prebuilt/windows-x86_64/bin/aarch64-linux-android-addr2line.exe' -e ./app/src/main/jniLibs/arm64-v8a/libmain.so -f -C
A notable advantage of this approach is that it doesn't require packaged / uploaded libraries to have symbols (so long as you can reference a copy of the library with symbols when symbolizing) and doesn't require the use of extractNativeLibs.
For example with this backtrace from adb:
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #00 pc 0000000000021d94 /system/lib64/libc.so (abort+124) 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #01 pc 000000000199ad18 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #02 pc 0000000001997b1c /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #03 pc 00000000019979c0 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #04 pc 00000000019976e8 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #05 pc 0000000001995a88 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #06 pc 0000000001997420 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #07 pc 000000000014b604 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #08 pc 000000000014b6a8 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #09 pc 0000000000950224 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #10 pc 0000000000b69ec0 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #11 pc 0000000000a1f490 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #12 pc 0000000000b64a24 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #13 pc 0000000000b599f0 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #14 pc 0000000000b594d8 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #15 pc 0000000000b64774 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #16 pc 000000000099f968 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #17 pc 0000000001617d68 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #18 pc 000000000014fa14 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #19 pc 000000000014be74 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so (android_main+144) 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #20 pc 0000000001966740 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so (_rust_glue_entry+348) 2022-05-10 04:54:22.928 29166-29166/? A/DEBUG: #21 pc 0000000001976654 /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
The above script will pull out:
0000000000021d94 000000000199ad18 0000000001997b1c 00000000019979c0 00000000019976e8 0000000001995a88 0000000001997420 000000000014b604 000000000014b6a8 0000000000950224 0000000000b69ec0 0000000000a1f490 0000000000b64a24 0000000000b599f0 0000000000b594d8 0000000000b64774 000000000099f968 0000000001617d68 000000000014fa14 000000000014be74 0000000001966740 0000000001976654
Which can be piped to addr2line to get:
?? ??:0 std::sys::unix::abort_internal::hf2e00ab422f03a90 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/sys/unix/mod.rs:259 rust_panic /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:748 std::panicking::rust_panic_with_hook::hcdc16bc1d6a90d9f /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:716 std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h80c5bec89bedcc17 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:588 std::sys_common::backtrace::__rust_end_short_backtrace::hbdfac0e0f8306365 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/sys_common/backtrace.rs:138 rust_begin_unwind /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:584 core::panicking::panic_fmt::h28a757e2227ef488 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/core/src/panicking.rs:143 core::result::unwrap_failed::hc5bc744b126af212 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/core/src/result.rs:1749 core::result::Result$LT$T$C$E$GT$::unwrap::hdee8c853a0eb3fec /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1065 bevy_render::renderer::initialize_renderer::_$u7b$$u7b$closure$u7d$$u7d$::h0cd0ec9c415b7d46 C:\Users\Robert\src\bevy\crates\bevy_render\src\renderer/mod.rs:224 _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hceb4b9af4b98638b /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84 futures_lite::future::block_on::_$u7b$$u7b$closure$u7d$$u7d$::hcea9b5092dbbef10 C:\Users\Robert\.cargo\registry\src\github.com-1ecc6299db9ec823\futures-lite-1.12.0\src/future.rs:89 std::thread::local::LocalKey$LT$T$GT$::try_with::h5f1204b6f3f53c68 /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:413 std::thread::local::LocalKey$LT$T$GT$::with::hce01bea0642e5b3f /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:389 futures_lite::future::block_on::hb3d920eab81bb4be C:\Users\Robert\.cargo\registry\src\github.com-1ecc6299db9ec823\futures-lite-1.12.0\src/future.rs:79 _$LT$bevy_render..RenderPlugin$u20$as$u20$bevy_app..plugin..Plugin$GT$::build::h6259e21f62b5446d C:\Users\Robert\src\bevy\crates\bevy_render\src/lib.rs:140 bevy_app::plugin_group::PluginGroupBuilder::finish::hc623480a2b4343d0 C:\Users\Robert\src\bevy\crates\bevy_app\src/plugin_group.rs:112 bevy_app::app::App::add_plugins::h860e64c350394fbd C:\Users\Robert\src\bevy\crates\bevy_app\src/app.rs:799 android_main C:\Users\Robert\src\agdk-rust\examples\agdk-bevy/src/lib.rs:38 _rust_glue_entry C:\Users\Robert\src\agdk-rust\game-activity\src/lib.rs:735 android_app_entry C:\Users\Robert\src\agdk-rust\game-activity/csrc/game-activity/native_app_glue/android_native_app_glue.c:208
Interesting how you managed to symbolize backtraces from logcat crash reports. I think that getting rust backtraces is a different issue. Many years ago before gimli became the default, backtraces did work, and they should work again.
Yeah, at the very least it would be good if backtrace-rs could determine valid library offsets - ideally without requiring legacy apk packing. I guess I could live without the on-device symbolizing considering the implicit need to package libraries with symbols which in turn makes packaging and uploading slower for testing.
Presumably for some versions of Android it can work with the extractNativeLibs workaround, or else maybe there has been a regression somewhere in backtrace-rs / gimli because for all the backtraces i've seen people posting for Android issues they all look to have bogus offsets.
Maybe a small thing but I think it's more awkward to scrape the system-provided backtrace from adb than it would be to scrape a Rust backtrace in situations where you're filtering the adb log for a single process (just because the system backtrace gets dumped by another system process).
