rust
rust copied to clipboard
rust-lang
macos: rustc built without jemalloc sporadically fails with realloc failure
Building rustc on macos without jemalloc (the default in config.toml) will trigger frequent SIGABRT. The error is:
rustc(14225,0x70000fab4000) malloc: *** error for object 0x600005c70800: pointer being realloc'd was not allocated
rustc(14225,0x70000fab4000) malloc: *** set a breakpoint in malloc_error_break to debug
The backtrace is:
Backtrace
* thread #2, name = 'rustc', stop reason = breakpoint 1.1
* frame #0: 0x00007ff80c53f5d7 libsystem_malloc.dylib`malloc_error_break
frame #1: 0x00007ff80c531376 libsystem_malloc.dylib`malloc_vreport + 440
frame #2: 0x00007ff80c5345ed libsystem_malloc.dylib`malloc_report + 151
frame #3: 0x00007ff80c525542 libsystem_malloc.dylib`realloc + 328
frame #4: 0x000000010c886c64 librustc_driver-cea70d83b3706378.dylib`_RINvNtCsk2H0Gr1Y1z1_5alloc7raw_vec11finish_growNtNtB4_5alloc6GlobalECsetNeWcxqv12_12rustc_expand + 52
frame #5: 0x000000010c88721d librustc_driver-cea70d83b3706378.dylib`_RNvMs_NtCsk2H0Gr1Y1z1_5alloc7raw_vecINtB4_6RawVecNtNtCsetNeWcxqv12_12rustc_expand3mbe9TokenTreeE16reserve_for_pushBP_ + 125
frame #6: 0x000000010c7f7222 librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe6quoted5parse + 4002
frame #7: 0x000000010c7f63fe librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe6quoted5parse + 382
frame #8: 0x000000010c8138eb librustc_driver-cea70d83b3706378.dylib`_RINvXs0_NtNtNtCsasXF9FuNFFy_4core4iter8adapters3mapINtB6_3MapINtNtNtBc_5slice4iter4IterNtNtNtCsetNeWcxqv12_12rustc_expand3mbe12macro_parser10NamedMatchENCNvNtB1r_11macro_rules25compile_declarative_macros_0ENtNtNtBa_6traits8iterator8Iterator4folduNCINvNvB3i_8for_each4callNtB1r_9TokenTreeNCNvXs_NtNtCsk2H0Gr1Y1z1_5alloc3vec11spec_extendINtB4K_3VecB4l_EINtB4I_10SpecExtendB4l_BN_E11spec_extend0E0EB1t_ + 267
frame #9: 0x000000010c861d47 librustc_driver-cea70d83b3706378.dylib`_RNvXNtNtCsk2H0Gr1Y1z1_5alloc3vec14spec_from_iterINtB4_3VecNtNtCsetNeWcxqv12_12rustc_expand3mbe9TokenTreeEINtB2_12SpecFromIterBU_INtNtNtNtCsasXF9FuNFFy_4core4iter8adapters3map3MapINtNtNtB2b_5slice4iter4IterNtNtBW_12macro_parser10NamedMatchENCNvNtBW_11macro_rules25compile_declarative_macros_0EE9from_iterBY_ + 247
frame #10: 0x000000010c8bd753 librustc_driver-cea70d83b3706378.dylib`_RNvNtNtCsetNeWcxqv12_12rustc_expand3mbe11macro_rules25compile_declarative_macro + 2451
frame #11: 0x000000010b75c904 librustc_driver-cea70d83b3706378.dylib`_RNvMs_NtCs9b1qYhXJ0Cf_13rustc_resolve6macrosNtB6_8Resolver13compile_macro + 68
frame #12: 0x000000010b70fd37 librustc_driver-cea70d83b3706378.dylib`_RNvMs3_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitor12define_macro + 295
frame #13: 0x000000010b710e08 librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 56
frame #14: 0x000000010b783a5c librustc_driver-cea70d83b3706378.dylib`_RINvNtCs9cCHMvxMcfg_9rustc_ast5visit9walk_itemNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEBM_ + 572
frame #15: 0x000000010b71304c librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 8828
frame #16: 0x000000010b783a5c librustc_driver-cea70d83b3706378.dylib`_RINvNtCs9cCHMvxMcfg_9rustc_ast5visit9walk_itemNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEBM_ + 572
frame #17: 0x000000010b71304c librustc_driver-cea70d83b3706378.dylib`_RNvXs4_NtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graphNtB5_24BuildReducedGraphVisitorNtNtCs9cCHMvxMcfg_9rustc_ast5visit7Visitor10visit_item + 8828
frame #18: 0x000000010b7d332d librustc_driver-cea70d83b3706378.dylib`_RINvMs6_NtCsetNeWcxqv12_12rustc_expand6expandNtB6_11AstFragment10visit_withNtNtCs9b1qYhXJ0Cf_13rustc_resolve19build_reduced_graph24BuildReducedGraphVisitorEB1f_ + 685
frame #19: 0x000000010b7519c7 librustc_driver-cea70d83b3706378.dylib`_RNvXNtCs9b1qYhXJ0Cf_13rustc_resolve6macrosNtB4_8ResolverNtNtCsetNeWcxqv12_12rustc_expand4base14ResolverExpand36visit_ast_fragment_with_placeholders + 487
frame #20: 0x000000010c7e0cc3 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander19collect_invocations + 707
frame #21: 0x000000010c7dbe80 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander21fully_expand_fragment + 256
frame #22: 0x000000010c7dbba2 librustc_driver-cea70d83b3706378.dylib`_RNvMs1_NtCsetNeWcxqv12_12rustc_expand6expandNtB5_13MacroExpander12expand_crate + 1186
frame #23: 0x0000000108c098bc librustc_driver-cea70d83b3706378.dylib`_RINvMNtCs3NuVQBhYW7C_13rustc_session5utilsNtNtB5_7session7Session4timeINtNtCsasXF9FuNFFy_4core6result6ResultNtNtCs9cCHMvxMcfg_9rustc_ast3ast5CrateNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvNtCsBlFisMo4u3_15rustc_interface6passes20configure_and_expands_0EB3a_ + 1116
frame #24: 0x0000000108b749c0 librustc_driver-cea70d83b3706378.dylib`_RNvNtCsBlFisMo4u3_15rustc_interface6passes20configure_and_expand + 528
frame #25: 0x0000000108b7a354 librustc_driver-cea70d83b3706378.dylib`_RNvMs0_NtCsBlFisMo4u3_15rustc_interface7queriesNtB5_7Queries9expansion + 836
frame #26: 0x0000000108a5520f librustc_driver-cea70d83b3706378.dylib`_RINvMs2_NtCsBlFisMo4u3_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0s0_0INtNtCsasXF9FuNFFy_4core6result6ResultINtNtB2d_6option6OptionNtB6_6LinkerENtCsdjh14Z7klAl_12rustc_errors13ErrorReportedEEB1m_ + 1327
frame #27: 0x0000000108a4aaa8 librustc_driver-cea70d83b3706378.dylib`_RINvCskTEkA9M7v6o_10rustc_span15with_source_mapINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCINvNtCsBlFisMo4u3_15rustc_interface9interface23create_compiler_and_runBJ_NCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0Es_0EB3n_ + 376
frame #28: 0x0000000108a5639f librustc_driver-cea70d83b3706378.dylib`_RINvMs_Cs7KVLOe8Wn3E_10scoped_tlsINtB5_9ScopedKeyNtCskTEkA9M7v6o_10rustc_span14SessionGlobalsE3setNCNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1H_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3G_E00B3G_EB57_ + 1311
frame #29: 0x0000000108a52ea2 librustc_driver-cea70d83b3706378.dylib`_RINvNtNtCs57FQuS03DEe_3std10sys_common9backtrace28___rust_begin_short_backtraceNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1m_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3l_E0B3l_EB4M_ + 146
frame #30: 0x0000000108a3fae5 librustc_driver-cea70d83b3706378.dylib`_RNSNvYNCINvMNtCs57FQuS03DEe_3std6threadNtBa_7Builder15spawn_uncheckedNCINvNtCsBlFisMo4u3_15rustc_interface4util51setup_callbacks_and_run_in_thread_pool_with_globalsNCINvNtB1c_9interface12run_compilerINtNtCsasXF9FuNFFy_4core6result6ResultuNtCsdjh14Z7klAl_12rustc_errors13ErrorReportedENCNvCsQmx6cSBOMI_12rustc_driver12run_compilers_0E0B3b_E0B3b_Es_0INtNtNtB3g_3ops8function6FnOnceuE9call_once6vtableB4C_ + 165
frame #31: 0x00000001007157c7 libstd-f20edb373511b023.dylib`std::sys::unix::thread::Thread::new::thread_start::h13e1d52aa9cad226 + 39
frame #32: 0x00007ff80c707514 libsystem_pthread.dylib`_pthread_start + 125
frame #33: 0x00007ff80c70302f libsystem_pthread.dylib`thread_start + 15
This happens almost 100% of the the time for me, though it can be sporadic.
Doing something like ./x.py build --stage=2 src/tools/cargo is enough to reproduce. It almost always fails while building core for stage2.
I've spent some time trying to debug it, but have not figured anything out.
Using jemalloc makes the error go away.
Currently using master (8ad3c1dd1d47f9ce7dfdf4a14c70c67e1790b0f5), though this has been going on for a while. If I get a chance, I'll try to bisect to see when it started. But for me, I think it started after I updated to Monterey (macOS 12.0).
I have XCode 13.1.
Other users have reported this at https://github.com/rust-lang/cargo/issues/10022.
More information about this is over at https://github.com/rust-lang/rust/issues/92185#issuecomment-999935682. These are likely the same issue, but I am not certain.
I also ran into this a while ago, reported it against rayon initially: https://github.com/rayon-rs/rayon/issues/901.
There was a non-Rust issue we found that reported similar issues: https://github.com/ibmdb/python-ibmdb/issues/648#issuecomment-959782632.
Analysis we've done so far: it seems that something is broken with the macOS 12-provided /usr/lib/libstdc++.6.dylib. Swapping to the homebrew-provided libstdc++ seems to fix the crash, and using link-cplusplus to swap to libc++ also seems to fix the crash. Switching to jemalloc or mimalloc seemed to fix this.
Does anyone have the ear of folks at Apple?
Does anyone have the ear of folks at Apple?
Yes, I've reached out to a friend.
@saethlin Has a radar been filed for this, and if so, what’s the radar number?
Hey folks, I'm the person who'd be closest to being in charge of the old libstdc++.dylib at Apple. For background information, we only ship libstdc++.dylib for backwards ABI compatibility with older applications that were linked against it years ago. If I'm not mistaken, we don't even ship it on Apple Silicon macs anymore.
The right fix would be to always use libc++, since that is (and has been for a while) the one supported C++ Standard Library on Apple platforms. Sorry if that sounds like a silly suggestion -- I don't have much context on this issue from your side. Is there a reason why you folks are using the older libstdc++.dylib?
I don't think it is linked to libstdc++.dylib
> otool -L rustc
rustc:
@rpath/librustc_driver-e04c6e4bb2b707be.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libstd-e0d1108e51bfb0d5.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
https://github.com/rust-lang/rust/issues/92173#issuecomment-1000363681 might have been discussing a different project?
I agree, this issue showed up in Rust built via Nix (https://github.com/NixOS/nixpkgs/issues/144704) and the build setup there uses libc++ on macOS.
Is this issue related to the crashes folks are observing in clippy on macOS in the new 1.59.0 release?
- https://github.com/rust-lang/rust-clippy/issues/8470
This issue here is about when rustc is built without jemalloc, whereas that issue is occurring with binaries downloaded via rustup and which presumably are using jemalloc.
@lopopolo Yea, it's the same issue.
@lilyball clippy is not built with jemalloc.
@ehuss should this be tagged as a stable-to-stable regression and up the priority? or is there another place I should be following along?
I'd leave it up to the clippy team to set the priority for fixing clippy specifically. Fixing for rustc is probably low priority, as the official builds use jemalloc. I'm not sure where to follow along. Here, or #92185 or on the clippy repo.
Some more details:
- It only happens for me when the block size is 256 bytes.
- Using malloc inception with
DYLD_INSERT_LIBRARIESand a module linked to the latestlibSystem.B, the issue does NOT occur. - It seems to be an issue with
freeas hooking justreallocstill triggered the issue, and as soon as I hooked free, the issue went away. - It does not occur when
MallocStackLoggingenv var is set either.
I just want to report that issue is currently blocking me in development. If there are any work around for the macos darwin tool chain and clippy errors... would love to hear about it.
Analysis we've done so far: it seems that something is broken with the macOS 12-provided /usr/lib/libstdc++.6.dylib. Swapping to the homebrew-provided libstdc++ seems to fix the crash, and using link-cplusplus to swap to libc++ also seems to fix the crash. Switching to jemalloc or mimalloc seemed to fix this.
I've been tracking this issue. My fix was to downgrade to my project's MSRV, 1.57. But after the latest "Command Line Tools for Xcode" update (13.3), I found that I could run clippy again on both stable-x86_64-apple-darwin and stable-aarch64-apple-darwin. I haven't found anything in the changelog that is directly related to this, but it fixed the problem for me, and I think this could be of use.
I am running into the issue myself. What seems to be a workaround for everybody?
The definitive fix is to update to Xcode 13.3, you may need to use the App Store or Settings App (I've had it come differently on two different machines)
I believe one of the malloc implementations shipped with the 'shared library' feature of Xcode was broken.
From my testing, I believe it's completely unrelated to the rust alloc code.
I got XCode 13.3 and removed my rustup and reinstalled my version (1.51). Do I need to do anything else special because it seems to not have done anything?
so I am unsure if it is the same issue but things were fine on big sir and then I upgrade. My particular project has an enormous amount of dependencies (>800). Occasionally when doing a cargo clean, cargo update some libraries fail for apparently no reason (SIGABORT). If I rerun multiple times things seem to eventually build.
error: could not compile `web3`
Caused by:
process didn't exit successfully: `rustc --crate-name web3 --edition=2018.......(signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...```
Sometimes its web3, sometimes in json-rpc, seems to be random packages.
Compiling rusoto_kms v0.45.0
error: could not compile `rusoto_kms`
Caused by:
process didn't exit successfully: `rustc --crate-name rusoto_kms --edition=2018```
Very verbose gives me no more info
You can run it with lldb to check the callstack.
Our issue was actually logging the malloc issue (Source of this ticket and clippy issue):
rustc(14225,0x70000fab4000) malloc: *** error for object 0x600005c70800: pointer being realloc'd was not allocated
rustc(14225,0x70000fab4000) malloc: *** set a breakpoint in malloc_error_break to debug
The above issue appears fixed as of Xcode 13.3.
Should I create a new ticket?
If possible, debug cargo/rustc and print the callstack, if it's different, then definitely!
lldb cargo check (or w/e crashes)
lldb:> process launch
lldb:> process continue
-> when it catches sigabrt
lldb:> thread backtrace
Compiling diesel v1.4.8
error: could not compile `cookie_store`
Caused by:
process didn't exit successfully: `rustc --crate-name cookie_store --edition=2018 ........ --cap-lints allow` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
Process 64899 stopped
* thread #5, stop reason = signal SIGUSR1
frame #0: 0x00007ff80dc1da76 libsystem_pthread.dylib`_pthread_cond_wait + 1256
libsystem_pthread.dylib`_pthread_cond_wait:
-> 0x7ff80dc1da76 <+1256>: callq 0x7ff80dc1a97e ; pthread_testcancel
0x7ff80dc1da7b <+1261>: leaq -0x88(%rbp), %rax
0x7ff80dc1da82 <+1268>: movq 0x10(%rax), %rax
0x7ff80dc1da86 <+1272>: movq %rax, 0x8(%rbx)
Target 0: (cargo) stopped.
(lldb)
Process 64899 resuming
Process 64899 stopped
* thread #5, stop reason = signal SIGUSR1
frame #0: 0x00007ff80dc1da86 libsystem_pthread.dylib`_pthread_cond_wait + 1272
libsystem_pthread.dylib`_pthread_cond_wait:
-> 0x7ff80dc1da86 <+1272>: movq %rax, 0x8(%rbx)
0x7ff80dc1da8a <+1276>: jmp 0x7ff80dc1daab ; <+1309>
0x7ff80dc1da8c <+1278>: movq -0x58(%rbp), %rcx
0x7ff80dc1da90 <+1282>: movq -0x40(%rbp), %rdi
Target 0: (cargo) stopped.
You would have to debug rustc directly I believe. It's kinda hard to explain over github. Are you in the discord/zulip?
What is the discord link? Like I said I don't know if that will help as when I run these individually they seem fine. Is there a way to get the rustc output when doing cargo b?
https://discord.com/invite/rust-lang
cargo build --verbose will print the full rustc command you can run in the same dir to produce the same output
Then you can run lldb on that rustc command.
rustc(69444,0x700006d18000) malloc: *** error for object 0x600002ff1b00: pointer being realloc'd was not allocated rustc(69444,0x700006d18000) malloc: *** set a breakpoint in malloc_error_break to debug Abort trap: 6```