rls
rls copied to clipboard
rust-lang
Linux memory usage regression
I noticed RLS can consume quite a lot of memory lately. Upstream: https://github.com/rust-lang/rust/issues/56980
Idle memory usage
Memory usage running rls --cli. Reading taken after running once until built, quitting and re-running again and waiting for build complete.
| Project | Linux nightly-2018-12-14 | Linux stable 1.31 |
|---|---|---|
| cargo init --bin | 80 MiB | 25 MiB |
| rust-random/rand | 500 MiB | 45 MiB |
| rust-lang/regex | 1.5 GiB | 81 MiB |
So current stable gives a decent idea of where it should be. I tested beta channel out of interest and had results similar to current nightly.
Status
Tests use regex d4b9419 on Arch Linux
2019-03-10
| Release | Idle memory usage |
|---|---|
| 1.31.1 | 62 MB |
| 1.32.0 | 1.6 GB |
| 1.33.0 | 800 MB |
Update 2019-07-04
| Release | Idle memory usage |
|---|---|
| 1.34.2 | 740 MB |
| 1.35.0 | 1.3 GB |
| 1.36.0 | 1.0 GB |
Update 2019-08-16
| Release | Idle memory usage |
|---|---|
| 1.31.1 | 83 MB |
| 1.37.0 | 2.4 GB |
Does stable include the removal of jemalloc?
Memory consumption on windows still seems to be reasonable:
80 MiB after building rust-lang/regex:
$ rustc --version
rustc 1.32.0-nightly (4a45578bc 2018-12-07)
$ rls --version
rls-preview 1.31.6 (bfa1371 2018-12-07)
Thanks for testing Windows, so perhaps it's not affected. I did try selecting system & jemalloc allocators but they both had similarly high usage for me on Linux.
I believe we build Rust with jemalloc (only libstd?), so the distribution artifacts (incl. RLS) should use jemalloc.
Since system allocator is being selected since recently, is it possible we use both allocators (system for locally-built RLS space and jemalloc for the dynamically linked rustc-land), leading to increased memory usage?
I bisected the memory regression based on nightly toolchain releases
Testing as above with the regex crate:
-
rustup run nightly-2018-11-03 rls --cli-> 100 MiB - nightly-2018-11-04, nightly-2018-11-05 rls is not available
-
rustup run nightly-2018-11-06 rls --cli-> 1.5 GiB
I can also deduce this regression is caused upstream if I can reproduce it using the same rls source + different compilers.
For example using current rls code (173be7729ae9ea41303fbd84f339f76554c9d9af),
plus a small fix for older compilers
diff --git src/build/rustc.rs src/build/rustc.rs
index be3c52c..9ebc925 100644
--- src/build/rustc.rs
+++ src/build/rustc.rs
@@ -301,6 +301,7 @@ impl<'a> CompilerCalls<'a> for RlsRustcCalls {
edition: match state.session.edition() {
RustcEdition::Edition2015 => Edition::Edition2015,
RustcEdition::Edition2018 => Edition::Edition2018,
+ _ => panic!(),
},
};
let files = fetch_input_files(state.session);
Again testing as above with the regex crate:
-
rustup run nightly-2018-11-03 cargo run --manifest-path ../rls/Cargo.toml --no-default-features --release -- --cli-> 100 MiB -
rustup run nightly-2018-11-06 cargo run --manifest-path ../rls/Cargo.toml --no-default-features --release -- --cli-> 1.6 GiB
With 1.32 coming out I thought I'd retest rls memory usage to provide some more recent numbers.
| Project | Linux stable 1.31 / nightly-2018-11-03 | Linux stable 1.32 |
|---|---|---|
| rust-lang/regex | 80 MiB | 1.4 GiB |
So this regression has not been addressed and will affect the next stable release.
Since switch to system as default allocator tools don't use jemalloc anymore: https://github.com/rust-lang/rust/issues/56980#issuecomment-455227996
I can reproduce huge memory usage that doesn't go down even with RLS built on the very up to date Linux distro (latest available GLIBC, kernel, toolchain, etc). This is pretty worrisome because it means there is leak somewhere or there is bug in memory handling. It was masked by jemalloc but that doesn't mean there was no issue.
At this point underlying issue should be identified and fixed. I'll try adding similar hack as in https://github.com/rust-lang/rust/pull/57287 but it shouldn't be considered as fix.
EDIT: Is there a way to reproduce large memory usage by manually running a rls command instead of using IDE? It's not easy to debug it when it's run from IDE.
@jens1o well I was hoping for something like rls arg1 arg2 arg3 ... and boom 600 MiB memory consumed 😄
I've tried forcing use of jemalloc & adding the stuff in https://github.com/rust-lang/rust/pull/57287 to rls main (though I don't understand it). These don't seem to help the memory usage regression though, or maybe I'm doing something wrong.
I don't have good news. After switching RLS to use jemmalloc it's peak memory usage increased a lot but later it drops to numbers similar to system allocator. Clippy and Cargo were still using system allocator during testing.
Valgrind and Heapstack didn't give any meaningful results. Maybe somebody more skilled could find out what's wrong, I'm out of ideas.
10GB used for me after initial build completed on https://salsa.debian.org/Kazan-team/kazan rls-preview 1.33.0 (9f3c906 2019-01-20) installed via rustup Linux x86_64
Note that building kazan builds LLVM as part of the process, so it can take a really long time and lots of memory.
VS Code + rls consume 3GB RAM, 1.4GB for rls itself. This is a huge issue on systems with 8GB RAM. I'll have to upgrade the RAM on my laptop because of this.
I just did Massif runs on rust-lang/regex, RLS itself and a cargo new project using latest RLS master built with rustc 1.35.0-nightly (53f2165c5 2019-04-04) and I can't reproduce it anymore.
Keep in mind that this is with all_targets set to true (so every bin/lib/test target is compiled and analyzed).
This is a simple output for run on RLS (some packages were cached but cargo needed to build some from scratch at the beginning):
MB
441.5^ #
| #
| @@@:@@# @
| @@@@@:@@# :@@@@@
| :@@@@@@:@@# @@@@@@@
| @@@@@@@@:@@# :@@@@@@@@
| @@@@@@@@:@@# @@@@@@@@@@
| @@:@@@@@@@@@@:@@# @:@@@@@@@@@@
| :@@:@@@@@@@@@@:@@# @@@:@@@@@@@@@@ @@@
| :@@@:@@@@@@@@@@:@@# :@@@:@@@@@@@@@@ @@@@
| :@@@:@@@@@@@@@@:@@# @@@@:@@@@@@@@@@ @@@@@ @@
| @@@@:@@@@@@@@@@:@@# @@@@:@@@@@@@@@@ @@@@@@ @@@
| :@@@@:@@@@@@@@@@:@@# @@@@@:@@@@@@@@@@ :@ @ @@@@@@ @@@
| :@:@@@@:@@@@@@@@@@:@@# @@@@@@:@@@@@@@@@@ @@ @ :@@@@@@ @@@
| : @@:@@@@:@@@@@@@@@@:@@# @@@@@@@:@@@@@@@@@@ @@:@ @@@@@@@@@@@
| : :@@:@@@@:@@@@@@@@@@:@@# @@@@@@@:@@@@@@@@@@:@@:@@@@@@@@@@@@@
| :: @@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@
| @:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@
| @:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@@ :@@@@
|@@:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@@@@@@@@
0 +----------------------------------------------------------------------->Gi
0 75.77
While it peaks at ~440MB, after everything is rebuilt it then stays on idle at 55MB.
For regex package it's slightly different (reopened folder, previously RLS built the dependencies):
MB
511.7^ #
| @#
| : @# :
| :: @# @:
| @: @@# @: ::@@ @@
| @: @@# @@@: @@:@@ @@ @@@@
| @@: : @@# @@@@: :@@:@@ @@@ @@@@@
| @@: : @@# :@@@@: @@@:@@ @@@@ @@@@@
| @@: @@@ @@# @@@@@: @@@@:@@ @@@@@ @@@@@@
| @@: @@@@ @@# @@@@@@: @@@@:@@ @@@@@ @@@@@@@
| @@: @@@@:@@# :@@@@@@: @@@@@@@:@@ @@@@@@@ @@@@@@@@@
| @@: @@@@@:@@# @@:@@@@@@: @@@@@@@@:@@@:@@@@@@@ @@@@@@@@@@
| : @@@: @: @@@@@:@@#:@@:@@@@@@: @@@@@@@@:@@@:@@@@@@@ @@@@@@@@@@:
| :@ @@@: :@: @@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@ @@@: @@::@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@:@@@@: :@@:@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@:@@@@::@@@:@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@:@@@@:@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@:@@@@:@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
|@@@:@@@@@@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
0 +----------------------------------------------------------------------->Gi
0 234.5
Then idle is 130 MB. What's interesting in this case is that there's
- 51MB (__rdos_backtrace_alloc)
->30.55% (53,281,105B) 0x8A58A16: __rdos_backtrace_alloc (alloc.c:57)
| ->24.96% (43,520,972B) 0x8A60433: __rdos_backtrace_get_view (read.c:61)
| | ->13.66% (23,815,387B) 0x8A618F0: elf_add (elf.c:2945)
| | | ->13.66% (23,815,387B) 0x8A62A5B: phdr_callback (elf.c:3270)
| | | ->13.66% (23,815,387B) 0x9D55F1F: dl_iterate_phdr (dl-iteratephdr.c:75)
| | | ->13.66% (23,815,387B) 0x8A60F61: __rdos_backtrace_initialize (elf.c:3313)
| | | ->13.66% (23,815,387B) 0x8A60295: fileline_initialize (fileline.c:143)
| | | ->13.66% (23,815,387B) 0x8A60306: __rdos_backtrace_syminfo (fileline.c:193)
| | | ->13.66% (23,815,387B) 0x8A43023: std::sys_common::backtrace::_print (libbacktrace.rs:71)
| | | ->13.66% (23,815,387B) 0x8A47105: std::panicking::default_hook::{{closure}} (backtrace.rs:59)
| | | ->13.66% (23,815,387B) 0x8A46E98: std::panicking::default_hook (panicking.rs:211)
| | | ->13.66% (23,815,387B) 0x713325E: rustc::util::common::panic_hook (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknow
n-linux-gnu/lib/librustc-4f0fde688aa35e5e.so)
| | | ->13.66% (23,815,387B) 0x8A478F7: std::panicking::rust_panic_with_hook (panicking.rs:478)
| | | ->13.66% (23,815,387B) 0x8A47390: std::panicking::continue_panic_fmt (panicking.rs:381)
| | | ->13.66% (23,815,387B) 0x8A47274: rust_begin_unwind (panicking.rs:308)
| | | ->13.66% (23,815,387B) 0x8A7010B: core::panicking::panic_fmt (panicking.rs:85)
| | | ->13.66% (23,815,387B) 0x8A7004A: core::panicking::panic (panicking.rs:49)
| | | ->13.66% (23,815,387B) 0x11951AE: <regex_syntax::hir::translate::TranslatorI as regex_syntax::ast::visitor::Visitor>::visit_post (macros.rs:12)
| | | ->13.66% (23,815,387B) 0x119E830: regex_syntax::ast::visitor::visit (visitor.rs:261)
| | | ->13.66% (23,815,387B) 0x117E1F4: regex_syntax::parser::Parser::parse (translate.rs:143)
| | | ->13.66% (23,815,387B) 0x661827: clippy_lints::regex::check_regex (regex.rs:216)
| | | ->13.66% (23,815,387B) 0x6610E3: <clippy_lints::regex::Pass as rustc::lint::LateLintPass>::check_expr (regex.rs:0)
| | | ->13.66% (23,815,387B) 0x6EAE501: <rustc::lint::context::LateLintPassObjects as rustc::lint::LateLintPass>::check_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc-4f0fde688aa35e5e.so)
| | | ->13.66% (23,815,387B) 0x529D840: <rustc::lint::context::LateContextAndPass<T> as rustc::hir::intravisit::Visitor>::visit_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x527FC67: rustc::hir::intravisit::walk_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x529D84B: <rustc::lint::context::LateContextAndPass<T> as rustc::hir::intravisit::Visitor>::visit_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x527FBAD: rustc::hir::intravisit::walk_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x529D84B: <rustc::lint::context::LateContextAndPass<T> as rustc::hir::intravisit::Visitor>::visit_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x529D84B: <rustc::lint::context::LateContextAndPass<T> as rustc::hir::intravisit::Visitor>::visit_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
| | | ->13.66% (23,815,387B) 0x527FC04: rustc::hir::intravisit::walk_expr (in /home/xanewok/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_interface-a592b137b81581d7.so)
- 51MB (__rbt_backtrace_alloc)
->35.79% (53,018,184B) 0x1063E80: __rbt_backtrace_alloc (in /home/xanewok/repos/rls/target/release/rls)
| ->29.37% (43,520,972B) 0x106AED2: __rbt_backtrace_get_view (in /home/xanewok/repos/rls/target/release/rls)
| | ->16.07% (23,815,387B) 0x1063294: elf_add (in /home/xanewok/repos/rls/target/release/rls)
| | | ->16.07% (23,815,387B) 0x1063C62: phdr_callback (in /home/xanewok/repos/rls/target/release/rls)
| | | ->16.07% (23,815,387B) 0x9D55F1F: dl_iterate_phdr (dl-iteratephdr.c:75)
| | | ->16.07% (23,815,387B) 0x1063DC1: __rbt_backtrace_initialize (in /home/xanewok/repos/rls/target/release/rls)
| | | ->16.07% (23,815,387B) 0x10605A9: __rbt_backtrace_pcinfo (in /home/xanewok/repos/rls/target/release/rls)
| | | ->16.07% (23,815,387B) 0x105FB38: backtrace::symbolize::resolve (libbacktrace.rs:171)
| | | ->16.07% (23,815,387B) 0x105E9B8: backtrace::capture::Backtrace::resolve (capture.rs:146)
| | | ->16.07% (23,815,387B) 0x105E03E: failure::backtrace::internal::InternalBacktrace::as_backtrace (internal.rs:61)
| | | ->16.07% (23,815,387B) 0x105E0AA: <&T as core::fmt::Debug>::fmt (mod.rs:131)
| | | ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | | ->16.07% (23,815,387B) 0x8A77312: core::fmt::Formatter::write_fmt (mod.rs:1457)
| | | ->16.07% (23,815,387B) 0x105DD57: <failure::error::Error as core::fmt::Debug>::fmt (mod.rs:199)
| | | ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | | ->16.07% (23,815,387B) 0x8A7626F: <core::fmt::Arguments as core::fmt::Debug>::fmt (mod.rs:417)
| | | ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | | ->16.07% (23,815,387B) 0x112CD7C: env_logger::fmt::Builder::build::{{closure}} (mod.rs:1279)
| | | ->16.07% (23,815,387B) 0x1128017: std::thread::local::LocalKey<T>::with (lib.rs:804)
| | | ->16.07% (23,815,387B) 0x1126DC0: <env_logger::Logger as log::Log>::log (lib.rs:775)
| | | ->16.07% (23,815,387B) 0x11BEBA9: log::__private_api_log (lib.rs:1232)
| | | ->16.07% (23,815,387B) 0x2C1D03: rls::build::cargo::run_cargo_ws (cargo.rs:271)
| | | ->16.07% (23,815,387B) 0x2BD098: rls::build::cargo::run_cargo (cargo.rs:137)
| | | ->16.07% (23,815,387B) 0x452D33: _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h34609
f02c4e16181E.llvm.1588280635495087468 (cargo.rs:62)
| | | ->16.07% (23,815,387B) 0x33D463: _ZN3std9panicking3try7do_call17hb8fcd0f3e48228acE.llvm.6379468967674043134 (mod.rs:470)
| | | ->16.07% (23,815,387B) 0x8A585D8: __rust_maybe_catch_panic (lib.rs:87)
| | | ->16.07% (23,815,387B) 0x4C9E29: <F as alloc::boxed::FnBox<A>>::call_box (panicking.rs:272)
| | | ->16.07% (23,815,387B) 0x8A5733C: std::sys::unix::thread::Thread::new::thread_start (boxed.rs:762)
| | | ->16.07% (23,815,387B) 0x97C06D9: start_thread (pthread_create.c:463)
| | | ->16.07% (23,815,387B) 0x9D1188D: clone (clone.S:95)
which make up around 100 of the total ~130MB.
So while we're not in the gigabyte order of magnitude during idle (at least I can't confirm it, still need to test some on-and-off development and see if there is any prominent leakage) we still can improve things.
@nnethercote would you have any tips or insights on how we could debug it better, seeing as you profiled a lot of the Rust compiler itself and as the author/maintainer(?) of Massif?
First, use massif-visualizer. It's a graphical viewer for massif data that is much better than ms_print.
Second, if you build your own version of Valgrind from trunk (or wait until 3.15 comes out, which I believe should be soon) Massif will get better, because it can now use debug info to get better stack traces. The Massif that comes with Valgrind 3.14 and earlier doesn't know about inlining, so any inlined frames aren't shown, and in Rust code there is usually a lot of inlining. See here for how to build your own Valgrind.
Third, if you build your own (or wait until 3.15) you can use the new and improved version of DHAT, which tracks peak memory usage as well as a bunch of other things relating to heap memory. I find its presentation of the stacks at peak memory a bit nicer, but it doesn't have the time-based graph, and in your case the graph is useful because there are multiple peaks.
Testing with regex d4b9419 I still get ~900MB usage after idle, so no change for me with rustc 1.35.0-nightly (8159f389f 2019-04-06).
I tested with regex & rls checked out next to each other. Run in regex dir:
cargo run --manifest-path ../rls/Cargo.toml --no-default-features --release -- --cli
What can be done about this? The bug has existed since mid-Decmeber and is P-high. By not fixing this, it seems like Linux is being treated like it doesn't exist.
I just recorded RLS at 2.2 GB (on a machine with 8 GB) in VS Code with no files open. I had worked on Rust files previously, but it had been at least half an hour since I had any open.
I am absolutely willing to help, but given that no one is even assigned to this, I'm losing faith. If someone can point me in the right direction, that would be great.
What changed between 2018-11-03 and 2018-11-06, as indicated in this issue? I presume that's a starting place.
@jhpratt Linux has the best overall support. This issue is just harder to debug.
What changed between 2018-11-03 and 2018-11-06, as indicated in this issue? I presume that's a starting place.
Rust changed the way it links jemalloc, there is more info in Rust issue: https://github.com/rust-lang/rust/issues/56980 But that change haven't introduced the issue, it only exposed it in official builds. Unofficial builds without jemalloc were affected before that change.
Without upstream progress I think we'll need #1307 which should address this as the compiler memory usage will die with the process.
Will this issue ever be solved? RLS makes my PC totally freeze within minutes from opening VSCode
@alexcrichton @alexheretic would you please release the solution in #1307 at least in linux releases because development with RLS on Linux is really a painful experience due to this memory leak bug. It becomes exponentially worse as the project grows.
@gophobic @gh67uyyghj In the meantime you can give a try to rust-analyzer. It's frequently updated and works much better than RLS (on my setup at least :p)
@fstephany thank you, I just installed rust-analyzer using the commands on their README but couldn't get it work, all I see in VSCode is the error message Couldn't start client Rust Analyzer Language Server :(
FWIW This is now bad enough that I've disabled RLS in VS Code due to the extremely frequent restarts necessary. Memory would rise to over a 1.5 GB within a matter of a minute or two. This is just developing the time crate, not anything terribly large.
In case tougher reproductions are needed: In projects using godot-rust I'm hitting 6 GB (often sending my machine into swap-freezes).
Could you try using nightly RLS and see if that’s still the issue?
On Fri, 15 May 2020 at 10:52, Fabian Keller [email protected] wrote:
In case tougher reproductions are needed: In projects using godot-rust https://github.com/GodotNativeTools/godot-rust I'm hitting 6 GB (often sending my machine into swap-freezes).
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rust-lang/rls/issues/1188#issuecomment-629116245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXTFXIW5UCGWCL3IYX42NDRRT7DVANCNFSM4GKNBBFQ .
I'm using stable, and the memory usage is still terrible. Maybe language server is just a bad idea for every language. Non-long-running shared library plugins would work better.
The fix hasn’t been backported, so stable and beta are not fixed and have the memory issue.
On Sat, 16 May 2020 at 21:50, buckle2000 [email protected] wrote:
I'm using stable, and the memory usage is still terrible.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rust-lang/rls/issues/1188#issuecomment-629697611, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXTFXND7JYG4SSLNHWNPTDRR3VBBANCNFSM4GKNBBFQ .
rls 1.41.0 (2cf84ba 2020-12-06)
rls 1.41.0 still alloc huge memory (about 4.5G, on a machine with 8 GB, ubuntu 18.04) .
