rust
rust copied to clipboard
Published
20 hours ago •
rust-lang
rust-lang
backtraces broken on the Android bot
PIE is now required by Android (#17437) but it breaks backtrace support on the Android bot. It may just need to be updated to the current NDK / Android versions.
I've locally enabled PIE for android and tested this with the latest NDK (r10d) generated for android-21 and an android emulator running the latest API (android-22), and it still fails with:
test [run-pass] run-pass/backtrace.rs ... FAILED
failures:
---- [run-pass] run-pass/backtrace.rs stdout ----
error: test run failed!
status: exit code: 101
command: x86_64-apple-darwin/test/run-pass/backtrace.stage2-arm-linux-androideabi
stdout:
------------------------------------------
------------------------------------------
stderr:
------------------------------------------
thread '<main>' panicked at 'bad output: thread '<main>' panicked at 'explicit panic', /Users/tamird/src/rust/src/test/run-pass/backtrace.rs:23
stack backtrace:
1: 0xb6c53da7 - <unknown>
2: 0xb6c5ea8b - <unknown>
3: 0xb6c0df37 - <unknown>
4: 0xb6f87fb7 - <unknown>
5: 0xb6f87d73 - <unknown>
6: 0xb6f8b913 - <unknown>
7: 0xb6d0dfe3 - <unknown>
8: 0xb6d0dfbb - <unknown>
9: 0xb6d0dfbb - <unknown>
10: 0xb6d0dfbb - <unknown>
11: 0xb6d0dfbb - <unknown>
12: 0xb6d0dfbb - <unknown>
13: 0xb6d0dfbb - <unknown>
14: 0xb6d0dfbb - <unknown>
15: 0xb6d0dfbb - <unknown>
16: 0xb6d0dfbb - <unknown>
17: 0xb6d0dfbb - <unknown>
18: 0xb6d0dfbb - <unknown>
19: 0xb6d0dfbb - <unknown>
20: 0xb6d0dfbb - <unknown>
21: 0xb6d0dfbb - <unknown>
22: 0xb6d0dfbb - <unknown>
23: 0xb6d0dfbb - <unknown>
24: 0xb6d0dfbb - <unknown>
25: 0xb6d0dfbb - <unknown>
26: 0xb6d0dfbb - <unknown>
27: 0xb6d0dfbb - <unknown>
28: 0xb6d0dfbb - <unknown>
29: 0xb6d0dfbb - <unknown>
30: 0xb6d0dfbb - <unknown>
31: 0xb6d0dfbb - <unknown>
32: 0xb6d0dfbb - <unknown>
33: 0xb6d0dfbb - <unknown>
34: 0xb6d0dfbb - <unknown>
35: 0xb6d0dfbb - <unknown>
36: 0xb6d0dfbb - <unknown>
37: 0xb6d0dfbb - <unknown>
38: 0xb6d0dfbb - <unknown>
39: 0xb6d0dfbb - <unknown>
40: 0xb6d0dfbb - <unknown>
41: 0xb6d0dfbb - <unknown>
42: 0xb6d0dfbb - <unknown>
43: 0xb6d0dfbb - <unknown>
44: 0xb6d0dfbb - <unknown>
45: 0xb6d0dfbb - <unknown>
46: 0xb6d0dfbb - <unknown>
47: 0xb6d0dfbb - <unknown>
48: 0xb6d0dfbb - <unknown>
49: 0xb6d0dfbb - <unknown>
50: 0xb6d0dfbb - <unknown>
51: 0xb6d0dfbb - <unknown>
52: 0xb6d0dfbb - <unknown>
53: 0xb6d0dfbb - <unknown>
54: 0xb6d0dfbb - <unknown>
55: 0xb6d0dfbb - <unknown>
56: 0xb6d0dfbb - <unknown>
57: 0xb6d0dfbb - <unknown>
58: 0xb6d0dfbb - <unknown>
59: 0xb6d0dfbb - <unknown>
60: 0xb6d0dfbb - <unknown>
61: 0xb6d0dfbb - <unknown>
62: 0xb6d0dfbb - <unknown>
63: 0xb6d0dfbb - <unknown>
64: 0xb6d0dfbb - <unknown>
65: 0xb6d0dfbb - <unknown>
66: 0xb6d0dfbb - <unknown>
67: 0xb6d0dfbb - <unknown>
68: 0xb6d0dfbb - <unknown>
69: 0xb6d0dfbb - <unknown>
70: 0xb6d0dfbb - <unknown>
71: 0xb6d0dfbb - <unknown>
72: 0xb6d0dfbb - <unknown>
73: 0xb6d0dfbb - <unknown>
74: 0xb6d0dfbb - <unknown>
75: 0xb6d0dfbb - <unknown>
76: 0xb6d0dfbb - <unknown>
77: 0xb6d0dfbb - <unknown>
78: 0xb6d0dfbb - <unknown>
79: 0xb6d0dfbb - <unknown>
80: 0xb6d0dfbb - <unknown>
81: 0xb6d0dfbb - <unknown>
82: 0xb6d0dfbb - <unknown>
83: 0xb6d0dfbb - <unknown>
84: 0xb6d0dfbb - <unknown>
85: 0xb6d0dfbb - <unknown>
86: 0xb6d0dfbb - <unknown>
87: 0xb6d0dfbb - <unknown>
88: 0xb6d0dfbb - <unknown>
89: 0xb6d0dfbb - <unknown>
90: 0xb6d0dfbb - <unknown>
91: 0xb6d0dfbb - <unknown>
92: 0xb6d0dfbb - <unknown>
93: 0xb6d0dfbb - <unknown>
94: 0xb6d0dfbb - <unknown>
95: 0xb6d0dfbb - <unknown>
96: 0xb6d0dfbb - <unknown>
97: 0xb6d0dfbb - <unknown>
98: 0xb6d0dfbb - <unknown>
99: 0xb6d0dfbb - <unknown>
100: 0xb6d0dfbb - <unknown>
... <frames omitted>
', /Users/tamird/src/rust/src/test/run-pass/backtrace.rs:54
------------------------------------------
thread '[run-pass] run-pass/backtrace.rs' panicked at 'explicit panic', /Users/tamird/src/rust/src/compiletest/runtest.rs:1525
Triage: we're in the process of moving all of the bots to Travis/AppVeyor, so I'm not sure if this 1. is still true 2. will persist there.
So after a bit of investigation, I found that android internally uses libunwind to handle its unwinding. It might be a good idea to try having rust use that ? If it's a good idea, I might start implementing it.
It's an option. We only used libunwind for a handful of releases. We've started writing our own unwinder since none of the off the shelf options were reliable or performant enough for our needs. That's getting close to ready, and we will be including it in the NDK.
@DanAlbert Oh, OK. So, if I make FFI bindings to libunwinderstack (when it's in the NDK) and use them in libstd, would I need to statically link to it ?
Either statically linked or included in the APK. The former is probably simpler.
Right. Thanks for the insights :D. I'll try figuring out how libunwinderstack works and hopefully draft a PR up over the next few days.
EDIT: Just found out libunwinderstack is written in CPP. Hopefully that won't hinder me too much, but it is going to be a bit more painful that I had anticipated at first.
So after some unfruitful attempts at making bindgen and libunwinderstack work together (I couldn't get bindgen to generate constructors for unwinderstack::Maps), I decided to investigate other options.
In doing so, I found out that backtrace-rs has no trouble getting a correct stacktrace. Furthermore, I figured that rust's tracing part seemed correct (when using RUST_BACKTRACE=full and comparing, the stack depth and pc seem correct), so the printing module seems to be at fault.
I'm going to be looking into the differences between backtrace-rs and the built-in backtracing of rust. I already have a few ideas.
I've been investigating this over the last few days and I think the issue is a missing dl_iterate_phdr function. It looks like that function was removed from android at some point.
Found this in the NDK which might be the issue:
#if __ANDROID_API__ >= 21
int dl_iterate_phdr(int (*)(struct dl_phdr_info*, size_t, void*), void*) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */
You're missing a small amount of scope there: that's wrapped in an #ifdef __arm__. It's more obvious what's going on if you look at the unprocessed source in bionic.
dl_iterate_phdr is not available for ARM Android (for whatever reason) until android-21 (Lollipop). I don't know the current state of libunwindstack, but the goal is for it to be usable for the NDK (so supporting ICS). If the dl_iterate_phdr call is coming from LLVM's libunwind, there was a patch that was submitted recently to add a fallback implementation if it's not available: https://reviews.llvm.org/D39468.
Thanks. I was trying to work out exactly where that function came from.
I was ignoring non-arm androids, as they are not that common atm.
Is there anything we need to do to test and then consume this fix? Or is just a case of waiting and re-enabling the tests once the change propagates to rust?
I was ignoring non-arm androids, as they are not that common atm.
idk about rust developers, but most of the time Android developers will use x86 emulators since they are much faster than an arm emulator.
Is there anything we need to do to test and then consume this fix? Or is just a case of waiting and re-enabling the tests once the change propagates to rust?
Should be automatic once you get the update. Do you periodically update from upstream, or do you track AOSP's copy (a fair number of projects do that for Android libraries) or something? If you're tracking AOSP lmk and I'll get it updated.
We have our own fork of AOSP that we need to bring back towards the official. We would need to bring any change into our own fork.
Okay, I'll try to find some time this week to get AOSP's unwinder updated so you at least get this fix when you sync up with upstream.
Two weeks late, but better than never: https://android-review.googlesource.com/c/platform/external/libunwind_llvm/+/567913
I Bumped into this issue myself today. I built the binary with this command:
cargo build --target armv7-linux-androideabi --release
Rust compiler version:
$ rustc --version
rustc 1.41.0 (5e1a79984 2020-01-27)
This is what the backtrace looks like:
I/flutter (22222): ⛔ main - stderr:
I/flutter (22222): thread '<unnamed>' panicked at 'internal error: entered unreachable code', components/funder/src/handler/canceler.rs:309:43
I/flutter (22222): stack backtrace:
I/flutter (22222): 0: <unknown>
I/flutter (22222): 1: <unknown>
I/flutter (22222): 2: <unknown>
I/flutter (22222): 3: <unknown>
I/flutter (22222): 4: <unknown>
I/flutter (22222): 5: <unknown>
I/flutter (22222): 6: <unknown>
I/flutter (22222): 7: <unknown>
I/flutter (22222): 8: <unknown>
I/flutter (22222): 9: <unknown>
I/flutter (22222): 10: <unknown>
I/flutter (22222): 11: <unknown>
I/flutter (22222): 12: <unknown>
I/flutter (22222): 13: <unknown>
I/flutter (22222): 14: <unknown>
I/flutter (22222): 15: <unknown>
I/flutter (22222): 16: <unknown>
I/flutter (22222): 17: <unknown>
I/flutter (22222): 18: <unknown>
I/flutter (22222): 19: <unknown>
I/flutter (22222):
I/flutter (22222): ⛔ main - stderr:
I/flutter (22222): 20: <unknown>
I/flutter (22222): 21: <unknown>
I/flutter (22222): note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
I/flutter (22222): [2020-03-17T12:50:00Z ERROR stcompact::server_loop] node() error: IndexClientError(AppServerClosed)
I/flutter (22222):
Latest output inside the emulator. This seems somewhat promising, as at least some symbols are showing up. It's not clear to me why we don't have them for the test case but do for std.
thread 'main' panicked at 'bad output: thread 'main' panicked at 'explicit panic', /checkout/src/test/ui/backtrace.rs:17:9
stack backtrace:
0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
1: core::fmt::write
2: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
3: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
4: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
5: std::panicking::rust_panic_with_hook
6: <unknown>
7: <unknown>
8: <unknown>
9: <unknown>
10: std::rt::lang_start_internal
11: <unknown>
12: __libc_init
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
', /checkout/src/test/ui/backtrace.rs:67:5
I'm encountering this issue on an Android 11 device... Any luck resolving it? I only get <unknown>.
backtrace-rs doesn't work either. unwind at least produces a list of addresses, but crashes with a segfault when it hits a bad address.
Hi, after many years, is there any solution? This is a critical issue, because stack traces are quite important to debug issues in production environments!
I need to at least know a list of addresses (and do not segfault even if hit bad addr!), such that I can copy these addresses and symbolize on my computer.
@s1341 Hi do you use it in production environment, such as --release, or only with debug? I do face problems sometimes there...
I use in release. What issues are you facing? Can you be more specific about when you have problems?
@s1341 Hi thanks for your reply! My issue: https://github.com/rust-lang/backtrace-rs/issues/445
Triage: The original issue is now 11 years old. I assume things have improved since then. Are backtraces still broken on Android? If yes, please provide detailed step by step instructions on how to reproduce. Thanks!
It turns out backtraces on Android are still broken, at least in terms of test failures. These tests fail when removing the ignore-android directives tied to this issue:
- tests/ui/backtrace/line-tables-only.rs
- tests/ui/backtrace/dylib-dep.rs
- tests/ui/backtrace/backtrace.rs
- tests/ui/panics/issue-47429-short-backtraces.rs
- tests/ui/panics/runtime-switch.rs
- tests/ui/panics/short-ice-remove-middle-frames-2.rs
- tests/ui/panics/short-ice-remove-middle-frames.rs
These tests don't fail, but I bet they would if the output was checked more closely in the tests:
- tests/ui/backtrace/std-backtrace.rs
- tests/ui/panics/location-detail-unwrap-multiline.rs
Click to expand to see full failure output
2025-09-27T15:40:51.3173030Z failures:
2025-09-27T15:40:51.3173197Z
2025-09-27T15:40:51.3173443Z ---- [ui] tests/ui/backtrace/line-tables-only.rs stdout ----
2025-09-27T15:40:51.3173805Z
2025-09-27T15:40:51.3183298Z error: test did not exit with success! code=Some(101) so test would pass with `run-fail`
2025-09-27T15:40:51.3183961Z status: exit status: 101
2025-09-27T15:40:51.3186124Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "1" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/line-tables-only/a" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/line-tables-only/auxiliary/libline_tables_only_helper.so"
2025-09-27T15:40:51.3188562Z --- stdout -------------------------------
2025-09-27T15:40:51.3189362Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/line-tables-only/a", waiting for result
2025-09-27T15:40:51.3190173Z ------------------------------------------
2025-09-27T15:40:51.3190577Z --- stderr -------------------------------
2025-09-27T15:40:51.3190957Z Backtrace [
2025-09-27T15:40:51.3191434Z { fn: "main", file: "./checkout/tests/ui/backtrace/line-tables-only.rs", line: 43 },
2025-09-27T15:40:51.3192579Z { fn: "call_once<fn(), ()>", file: "./rustc/FAKE_PREFIX/library/core/src/ops/function.rs", line: 250 },
2025-09-27T15:40:51.3193535Z { fn: "__rust_begin_short_backtrace<fn(), ()>", file: "./rustc/FAKE_PREFIX/library/std/src/sys/backtrace.rs", line: 158 },
2025-09-27T15:40:51.3194486Z { fn: "std::rt::lang_start::{{closure}}", file: "./rustc/FAKE_PREFIX/library/std/src/rt.rs", line: 206 },
2025-09-27T15:40:51.3195100Z { fn: "main" },
2025-09-27T15:40:51.3195388Z ]
2025-09-27T15:40:51.3195538Z
2025-09-27T15:40:51.3195892Z thread 'main' (2234) panicked at /checkout/tests/ui/backtrace/line-tables-only.rs:36:5:
2025-09-27T15:40:51.3196557Z backtrace does not contain expected name foo
2025-09-27T15:40:51.3196969Z stack backtrace:
2025-09-27T15:40:51.3197481Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3198092Z ------------------------------------------
2025-09-27T15:40:51.3198364Z
2025-09-27T15:40:51.3198615Z ---- [ui] tests/ui/backtrace/line-tables-only.rs stdout end ----
2025-09-27T15:40:51.3199180Z ---- [ui] tests/ui/backtrace/dylib-dep.rs stdout ----
2025-09-27T15:40:51.3199491Z
2025-09-27T15:40:51.3199888Z error: test did not exit with success! code=Some(134) so test would pass with `run-crash`
2025-09-27T15:40:51.3200496Z status: exit status: 134
2025-09-27T15:40:51.3203484Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "2" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/dylib-dep/a" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/dylib-dep/auxiliary/libdylib_dep_helper_aux.so" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/dylib-dep/auxiliary/libdylib_dep_helper.so"
2025-09-27T15:40:51.3206211Z --- stdout -------------------------------
2025-09-27T15:40:51.3206952Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/dylib-dep/a", waiting for result
2025-09-27T15:40:51.3207685Z died due to signal 6
2025-09-27T15:40:51.3208017Z ------------------------------------------
2025-09-27T15:40:51.3208423Z --- stderr -------------------------------
2025-09-27T15:40:51.3208817Z -----------------------------------
2025-09-27T15:40:51.3209169Z looking for:
2025-09-27T15:40:51.3209512Z /checkout/tests/ui/backtrace/dylib-dep.rs:74
2025-09-27T15:40:51.3210059Z /checkout/tests/ui/backtrace/auxiliary/dylib-dep-helper.rs:18
2025-09-27T15:40:51.3210750Z /checkout/tests/ui/backtrace/dylib-dep.rs:73
2025-09-27T15:40:51.3211153Z found:
2025-09-27T15:40:51.3211415Z Backtrace [
2025-09-27T15:40:51.3211898Z { fn: "dylib_dep::verify", file: "./checkout/tests/ui/backtrace/dylib-dep.rs", line: 43 },
2025-09-27T15:40:51.3212918Z { fn: "dylib_dep::main::{{closure}}", file: "./checkout/tests/ui/backtrace/dylib-dep.rs", line: 74 },
2025-09-27T15:40:51.3213873Z { fn: "core::ops::function::FnOnce::call_once", file: "./rustc/FAKE_PREFIX/library/core/src/ops/function.rs", line: 250 },
2025-09-27T15:40:51.3214809Z { fn: "dylib_dep::main", file: "./checkout/tests/ui/backtrace/dylib-dep.rs", line: 73 },
2025-09-27T15:40:51.3215716Z { fn: "core::ops::function::FnOnce::call_once", file: "./rustc/FAKE_PREFIX/library/core/src/ops/function.rs", line: 250 },
2025-09-27T15:40:51.3216836Z { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "./rustc/FAKE_PREFIX/library/std/src/sys/backtrace.rs", line: 158 },
2025-09-27T15:40:51.3217956Z { fn: "std::rt::lang_start::{{closure}}", file: "./rustc/FAKE_PREFIX/library/std/src/rt.rs", line: 206 },
2025-09-27T15:40:51.3218783Z { fn: "std::rt::lang_start", file: "./rustc/FAKE_PREFIX/library/std/src/rt.rs", line: 205 },
2025-09-27T15:40:51.3219397Z { fn: "main" },
2025-09-27T15:40:51.3219706Z ]
2025-09-27T15:40:51.3219866Z
2025-09-27T15:40:51.3220198Z thread 'main' (2231) panicked at /checkout/tests/ui/backtrace/dylib-dep.rs:57:9:
2025-09-27T15:40:51.3221250Z expected backtrace to contain /checkout/tests/ui/backtrace/auxiliary/dylib-dep-helper.rs
2025-09-27T15:40:51.3221983Z stack backtrace:
2025-09-27T15:40:51.3222712Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3223292Z
2025-09-27T15:40:51.3223617Z thread 'main' (2231) panicked at library/core/src/panicking.rs:228:5:
2025-09-27T15:40:51.3224285Z panic in a function that cannot unwind
2025-09-27T15:40:51.3224764Z stack backtrace:
2025-09-27T15:40:51.3225105Z 0: 0xb6e03cd4 - <unknown>
2025-09-27T15:40:51.3225426Z 1: 0xb6e51844 - <unknown>
2025-09-27T15:40:51.3225659Z 2: 0xb6d932d8 - <unknown>
2025-09-27T15:40:51.3225900Z 3: 0xb6dac3b8 - <unknown>
2025-09-27T15:40:51.3226136Z 4: 0xb6db61d0 - <unknown>
2025-09-27T15:40:51.3226365Z 5: 0xb6db5ee0 - <unknown>
2025-09-27T15:40:51.3226589Z 6: 0xb6db6a68 - <unknown>
2025-09-27T15:40:51.3226815Z 7: 0xb6db66ec - <unknown>
2025-09-27T15:40:51.3227048Z 8: 0xb6dac57c - <unknown>
2025-09-27T15:40:51.3227270Z 9: 0xb6d81da8 - <unknown>
2025-09-27T15:40:51.3227494Z 10: 0xb6e653d8 - <unknown>
2025-09-27T15:40:51.3227740Z 11: 0xb6e65264 - <unknown>
2025-09-27T15:40:51.3227969Z 12: 0xb6e6553c - <unknown>
2025-09-27T15:40:51.3228193Z 13: 0xb6e8a5fc - <unknown>
2025-09-27T15:40:51.3228471Z 14: 0xb6f47a50 - dylib_dep::main::ha3017fb5e5b597b8
2025-09-27T15:40:51.3228830Z at /checkout/tests/ui/backtrace/dylib-dep.rs:73:9
2025-09-27T15:40:51.3229357Z 15: 0xb6f43978 - core::ops::function::FnOnce::call_once::h202ac8eabc880af9
2025-09-27T15:40:51.3229796Z at /rustc/FAKE_PREFIX/library/core/src/ops/function.rs:250:5
2025-09-27T15:40:51.3230255Z 16: 0xb6f4331c - std::sys::backtrace::__rust_begin_short_backtrace::hb22deb6cb67257e2
2025-09-27T15:40:51.3230706Z at /rustc/FAKE_PREFIX/library/std/src/sys/backtrace.rs:158:18
2025-09-27T15:40:51.3231113Z 17: 0xb6f43088 - std::rt::lang_start::{{closure}}::h7f98b7106ce2b1fa
2025-09-27T15:40:51.3231484Z at /rustc/FAKE_PREFIX/library/std/src/rt.rs:206:18
2025-09-27T15:40:51.3231785Z 18: 0xb6d97960 - <unknown>
2025-09-27T15:40:51.3232063Z 19: 0xb6f43064 - std::rt::lang_start::hf0a2d6deffb60a03
2025-09-27T15:40:51.3232637Z at /rustc/FAKE_PREFIX/library/std/src/rt.rs:205:5
2025-09-27T15:40:51.3232932Z 20: 0xb6f487a8 - main
2025-09-27T15:40:51.3233151Z 21: 0xb6ec48c2 - <unknown>
2025-09-27T15:40:51.3233413Z thread caused non-unwinding panic. aborting.
2025-09-27T15:40:51.3233803Z ------------------------------------------
2025-09-27T15:40:51.3233980Z
2025-09-27T15:40:51.3234125Z ---- [ui] tests/ui/backtrace/dylib-dep.rs stdout end ----
2025-09-27T15:40:51.3234467Z ---- [ui] tests/ui/backtrace/backtrace.rs stdout ----
2025-09-27T15:40:51.3234667Z
2025-09-27T15:40:51.3234944Z error: test did not exit with success! code=Some(101) so test would pass with `run-fail`
2025-09-27T15:40:51.3235366Z status: exit status: 101
2025-09-27T15:40:51.3236340Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/backtrace/a"
2025-09-27T15:40:51.3237336Z --- stdout -------------------------------
2025-09-27T15:40:51.3237818Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/backtrace/backtrace/a", waiting for result
2025-09-27T15:40:51.3238302Z ------------------------------------------
2025-09-27T15:40:51.3238638Z --- stderr -------------------------------
2025-09-27T15:40:51.3238815Z
2025-09-27T15:40:51.3239026Z thread 'main' (2241) panicked at /checkout/tests/ui/backtrace/backtrace.rs:68:5:
2025-09-27T15:40:51.3239392Z bad output:
2025-09-27T15:40:51.3239698Z thread 'main' (2247) panicked at /checkout/tests/ui/backtrace/backtrace.rs:17:9:
2025-09-27T15:40:51.3240120Z explicit panic
2025-09-27T15:40:51.3240313Z stack backtrace:
2025-09-27T15:40:51.3240650Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3240951Z
2025-09-27T15:40:51.3241037Z stack backtrace:
2025-09-27T15:40:51.3241365Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3241755Z ------------------------------------------
2025-09-27T15:40:51.3241925Z
2025-09-27T15:40:51.3242070Z ---- [ui] tests/ui/backtrace/backtrace.rs stdout end ----
2025-09-27T15:40:51.3242678Z ---- [ui] tests/ui/panics/issue-47429-short-backtraces.rs stdout ----
2025-09-27T15:40:51.3243435Z Saved the actual run.stderr to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/issue-47429-short-backtraces/issue-47429-short-backtraces.run.stderr`
2025-09-27T15:40:51.3244089Z diff of run.stderr:
2025-09-27T15:40:51.3244213Z
2025-09-27T15:40:51.3244412Z 2 thread 'main' ($TID) panicked at $DIR/issue-47429-short-backtraces.rs:26:5:
2025-09-27T15:40:51.3244768Z 3 explicit panic
2025-09-27T15:40:51.3244960Z 4 stack backtrace:
2025-09-27T15:40:51.3245185Z - 0: std::panicking::begin_panic
2025-09-27T15:40:51.3245464Z - 1: issue_47429_short_backtraces::main
2025-09-27T15:40:51.3245863Z 7 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3246238Z 8
2025-09-27T15:40:51.3246342Z
2025-09-27T15:40:51.3246348Z
2025-09-27T15:40:51.3246504Z The actual run.stderr differed from the expected run.stderr
2025-09-27T15:40:51.3246745Z
2025-09-27T15:40:51.3246969Z error: 1 errors occurred comparing run output.
2025-09-27T15:40:51.3247260Z status: exit status: 101
2025-09-27T15:40:51.3248337Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="1" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/issue-47429-short-backtraces/a"
2025-09-27T15:40:51.3249432Z --- stdout -------------------------------
2025-09-27T15:40:51.3249968Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/issue-47429-short-backtraces/a", waiting for result
2025-09-27T15:40:51.3250508Z ------------------------------------------
2025-09-27T15:40:51.3250770Z --- stderr -------------------------------
2025-09-27T15:40:51.3250939Z
2025-09-27T15:40:51.3251208Z thread 'main' (10780) panicked at /checkout/tests/ui/panics/issue-47429-short-backtraces.rs:26:5:
2025-09-27T15:40:51.3251632Z explicit panic
2025-09-27T15:40:51.3251894Z stack backtrace:
2025-09-27T15:40:51.3252397Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3252804Z ------------------------------------------
2025-09-27T15:40:51.3252977Z
2025-09-27T15:40:51.3253170Z ---- [ui] tests/ui/panics/issue-47429-short-backtraces.rs stdout end ----
2025-09-27T15:40:51.3253577Z ---- [ui] tests/ui/panics/runtime-switch.rs stdout ----
2025-09-27T15:40:51.3254182Z Saved the actual run.stderr to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/runtime-switch/runtime-switch.run.stderr`
2025-09-27T15:40:51.3254737Z diff of run.stderr:
2025-09-27T15:40:51.3254866Z
2025-09-27T15:40:51.3255027Z 2 thread 'main' ($TID) panicked at $DIR/runtime-switch.rs:30:5:
2025-09-27T15:40:51.3255349Z 3 explicit panic
2025-09-27T15:40:51.3255552Z 4 stack backtrace:
2025-09-27T15:40:51.3255773Z - 0: std::panicking::begin_panic
2025-09-27T15:40:51.3256035Z - 1: runtime_switch::main
2025-09-27T15:40:51.3256481Z 7 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3256865Z 8
2025-09-27T15:40:51.3256967Z
2025-09-27T15:40:51.3256973Z
2025-09-27T15:40:51.3257127Z The actual run.stderr differed from the expected run.stderr
2025-09-27T15:40:51.3257365Z
2025-09-27T15:40:51.3257501Z error: 1 errors occurred comparing run output.
2025-09-27T15:40:51.3257794Z status: exit status: 101
2025-09-27T15:40:51.3258877Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="0" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/runtime-switch/a"
2025-09-27T15:40:51.3259923Z --- stdout -------------------------------
2025-09-27T15:40:51.3260520Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/runtime-switch/a", waiting for result
2025-09-27T15:40:51.3261099Z ------------------------------------------
2025-09-27T15:40:51.3261376Z --- stderr -------------------------------
2025-09-27T15:40:51.3261552Z
2025-09-27T15:40:51.3261766Z thread 'main' (11246) panicked at /checkout/tests/ui/panics/runtime-switch.rs:30:5:
2025-09-27T15:40:51.3262162Z explicit panic
2025-09-27T15:40:51.3262652Z stack backtrace:
2025-09-27T15:40:51.3263257Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3263998Z ------------------------------------------
2025-09-27T15:40:51.3264326Z
2025-09-27T15:40:51.3264595Z ---- [ui] tests/ui/panics/runtime-switch.rs stdout end ----
2025-09-27T15:40:51.3265170Z ---- [ui] tests/ui/panics/short-ice-remove-middle-frames-2.rs stdout ----
2025-09-27T15:40:51.3266459Z Saved the actual run.stderr to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames-2/short-ice-remove-middle-frames-2.run.stderr`
2025-09-27T15:40:51.3267167Z diff of run.stderr:
2025-09-27T15:40:51.3267297Z
2025-09-27T15:40:51.3267619Z 2 thread 'main' ($TID) panicked at $DIR/short-ice-remove-middle-frames-2.rs:61:5:
2025-09-27T15:40:51.3267999Z 3 debug!!!
2025-09-27T15:40:51.3268184Z 4 stack backtrace:
2025-09-27T15:40:51.3268410Z - 0: std::panicking::begin_panic
2025-09-27T15:40:51.3268692Z - 1: short_ice_remove_middle_frames_2::eight
2025-09-27T15:40:51.3269024Z - 2: short_ice_remove_middle_frames_2::seven::{{closure}}
2025-09-27T15:40:51.3269333Z - [... omitted 3 frames ...]
2025-09-27T15:40:51.3269602Z - 3: short_ice_remove_middle_frames_2::fifth
2025-09-27T15:40:51.3269923Z - 4: short_ice_remove_middle_frames_2::fourth::{{closure}}
2025-09-27T15:40:51.3270365Z + 0: short_ice_remove_middle_frames_2::fifth
2025-09-27T15:40:51.3270768Z + 1: short_ice_remove_middle_frames_2::fourth::{{closure}}
2025-09-27T15:40:51.3271074Z 11 [... omitted 4 frames ...]
2025-09-27T15:40:51.3271336Z - 5: short_ice_remove_middle_frames_2::first
2025-09-27T15:40:51.3271629Z - 6: short_ice_remove_middle_frames_2::main
2025-09-27T15:40:51.3272007Z - 7: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3272505Z + 2: short_ice_remove_middle_frames_2::first
2025-09-27T15:40:51.3272796Z + 3: short_ice_remove_middle_frames_2::main
2025-09-27T15:40:51.3273079Z + 4: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3273481Z 15 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3273866Z 16
2025-09-27T15:40:51.3273961Z
2025-09-27T15:40:51.3273967Z
2025-09-27T15:40:51.3274129Z The actual run.stderr differed from the expected run.stderr
2025-09-27T15:40:51.3274362Z
2025-09-27T15:40:51.3274520Z error: 1 errors occurred comparing run output.
2025-09-27T15:40:51.3274809Z status: exit status: 101
2025-09-27T15:40:51.3275972Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="1" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames-2/a"
2025-09-27T15:40:51.3277563Z --- stdout -------------------------------
2025-09-27T15:40:51.3278513Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames-2/a", waiting for result
2025-09-27T15:40:51.3279546Z ------------------------------------------
2025-09-27T15:40:51.3280000Z --- stderr -------------------------------
2025-09-27T15:40:51.3280445Z
2025-09-27T15:40:51.3280930Z thread 'main' (11250) panicked at /checkout/tests/ui/panics/short-ice-remove-middle-frames-2.rs:61:5:
2025-09-27T15:40:51.3281679Z debug!!!
2025-09-27T15:40:51.3281957Z stack backtrace:
2025-09-27T15:40:51.3282463Z 0: short_ice_remove_middle_frames_2::fifth
2025-09-27T15:40:51.3282992Z 1: short_ice_remove_middle_frames_2::fourth::{{closure}}
2025-09-27T15:40:51.3283485Z [... omitted 4 frames ...]
2025-09-27T15:40:51.3283918Z 2: short_ice_remove_middle_frames_2::first
2025-09-27T15:40:51.3284400Z 3: short_ice_remove_middle_frames_2::main
2025-09-27T15:40:51.3284876Z 4: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3285506Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3286149Z ------------------------------------------
2025-09-27T15:40:51.3286432Z
2025-09-27T15:40:51.3286764Z ---- [ui] tests/ui/panics/short-ice-remove-middle-frames-2.rs stdout end ----
2025-09-27T15:40:51.3287523Z ---- [ui] tests/ui/panics/short-ice-remove-middle-frames.rs stdout ----
2025-09-27T15:40:51.3288813Z Saved the actual run.stderr to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames/short-ice-remove-middle-frames.run.stderr`
2025-09-27T15:40:51.3289909Z diff of run.stderr:
2025-09-27T15:40:51.3290110Z
2025-09-27T15:40:51.3290422Z 2 thread 'main' ($TID) panicked at $DIR/short-ice-remove-middle-frames.rs:57:5:
2025-09-27T15:40:51.3290990Z 3 debug!!!
2025-09-27T15:40:51.3291272Z 4 stack backtrace:
2025-09-27T15:40:51.3291713Z - 0: std::panicking::begin_panic
2025-09-27T15:40:51.3292159Z - 1: short_ice_remove_middle_frames::seven
2025-09-27T15:40:51.3292774Z - 2: short_ice_remove_middle_frames::sixth
2025-09-27T15:40:51.3293268Z - 3: short_ice_remove_middle_frames::fifth::{{closure}}
2025-09-27T15:40:51.3293744Z - [... omitted 4 frames ...]
2025-09-27T15:40:51.3294168Z - 4: short_ice_remove_middle_frames::second
2025-09-27T15:40:51.3294709Z - 5: short_ice_remove_middle_frames::first::{{closure}}
2025-09-27T15:40:51.3295305Z - 6: short_ice_remove_middle_frames::first
2025-09-27T15:40:51.3295832Z - 7: short_ice_remove_middle_frames::main
2025-09-27T15:40:51.3296361Z - 8: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3296898Z + 0: short_ice_remove_middle_frames::second
2025-09-27T15:40:51.3297474Z + 1: short_ice_remove_middle_frames::first::{{closure}}
2025-09-27T15:40:51.3298046Z + 2: short_ice_remove_middle_frames::first
2025-09-27T15:40:51.3298578Z + 3: short_ice_remove_middle_frames::main
2025-09-27T15:40:51.3299204Z + 4: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3299971Z 15 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3300650Z 16
2025-09-27T15:40:51.3300820Z
2025-09-27T15:40:51.3300830Z
2025-09-27T15:40:51.3301120Z The actual run.stderr differed from the expected run.stderr
2025-09-27T15:40:51.3301559Z
2025-09-27T15:40:51.3301840Z error: 1 errors occurred comparing run output.
2025-09-27T15:40:51.3302534Z status: exit status: 101
2025-09-27T15:40:51.3304356Z command: RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="1" RUST_TEST_THREADS="4" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames/a"
2025-09-27T15:40:51.3306421Z --- stdout -------------------------------
2025-09-27T15:40:51.3307582Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panics/short-ice-remove-middle-frames/a", waiting for result
2025-09-27T15:40:51.3308605Z ------------------------------------------
2025-09-27T15:40:51.3309095Z --- stderr -------------------------------
2025-09-27T15:40:51.3309420Z
2025-09-27T15:40:51.3309949Z thread 'main' (11251) panicked at /checkout/tests/ui/panics/short-ice-remove-middle-frames.rs:57:5:
2025-09-27T15:40:51.3310862Z debug!!!
2025-09-27T15:40:51.3311169Z stack backtrace:
2025-09-27T15:40:51.3311569Z 0: short_ice_remove_middle_frames::second
2025-09-27T15:40:51.3312140Z 1: short_ice_remove_middle_frames::first::{{closure}}
2025-09-27T15:40:51.3312881Z 2: short_ice_remove_middle_frames::first
2025-09-27T15:40:51.3313413Z 3: short_ice_remove_middle_frames::main
2025-09-27T15:40:51.3313937Z 4: core::ops::function::FnOnce::call_once
2025-09-27T15:40:51.3314827Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2025-09-27T15:40:51.3315613Z ------------------------------------------
2025-09-27T15:40:51.3315915Z
2025-09-27T15:40:51.3316222Z ---- [ui] tests/ui/panics/short-ice-remove-middle-frames.rs stdout end ----
2025-09-27T15:40:51.3316677Z
2025-09-27T15:40:51.3316817Z failures:
2025-09-27T15:40:51.3317198Z [ui] tests/ui/backtrace/line-tables-only.rs
2025-09-27T15:40:51.3317715Z [ui] tests/ui/backtrace/dylib-dep.rs
2025-09-27T15:40:51.3318161Z [ui] tests/ui/backtrace/backtrace.rs
2025-09-27T15:40:51.3318713Z [ui] tests/ui/panics/issue-47429-short-backtraces.rs
2025-09-27T15:40:51.3319269Z [ui] tests/ui/panics/runtime-switch.rs
2025-09-27T15:40:51.3319846Z [ui] tests/ui/panics/short-ice-remove-middle-frames-2.rs
2025-09-27T15:40:51.3320477Z [ui] tests/ui/panics/short-ice-remove-middle-frames.rs
2025-09-27T15:40:51.3320823Z
2025-09-27T15:40:51.3321228Z test result: FAILED. 19407 passed; 7 failed; 432 ignored; 0 measured; 0 filtered out; finished in 952.55s