[Important]Memory Leak

[Percivalll]

Crate Version:0.14.5-Latest Kernel Version:Linux 4.14.81.bm.15-amd64 #1 SMP Debian 4.14.81.bm.15 Sun Sep 8 05:02:31 UTC 2019 x86_64 GNU/Linux Recurrence:

use sysinfo::{System,SystemExt};
fn main() {
    loop {
        let mut system = sysinfo::System::new_all();
        system.refresh_all();
        system.refresh_cpu();
        system.get_total_memory();
        system.get_free_memory();
        system.get_used_memory();
        system.get_used_swap();
        system.get_free_swap();
        system.get_used_swap();
        system.get_processors();
        system.get_disks();
    }
}

Then,watch you RSS of this process.Woooooooooow,you will get a big surprise!!!!

[GuillaumeGomez]

You're not supposed to create System more than once. However this is intriguing, thanks for opening this issue!

[Percivalll]

You're not supposed to create System more than once. However this is intriguing, thanks for opening this issue!

Emmmmmm,I mean when the System leaves its scope, its resources should be completely released, right?

[GuillaumeGomez]

Yes it should, it's just that the issue isn't that critical considering that you're only supposed to have one System. I'll still try to fix the memory leak.

[Percivalll]

Yes it should, it's just that the issue isn't that critical considering that you're only supposed to have one System. I'll still try to fix the memory leak.

Okay, thank you for your contribution to this project, and hope to see the new version of the fix soon!

[GuillaumeGomez]

Sorry but I really don't understand how the RSS is getting so high. I limited it to for _ in range 0..100 and I get a RSS of 322MB whereas the peak memory usage remains to 4MB (I got those numbers with heaptrack). So at this point, if someone has pointers to where it could be, I'm open to ideas!

[Percivalll]

I also observed this phenomenon,and I got the same result using heaptrack.In the future, I will pay attention to this problem as much as possible, if I find a solution, I will mention a PR to help fix it.

[GuillaumeGomez]

Thanks!

[EccoTheFlintstone]

I tried the aforementioned code in a loop but didn't see any leak (memory was steady at 1 GB) on a linux box

What I saw was a heavy memory consumption when refreshing processes :

let mut system = sysinfo::System::new();  // don't use new_all not to refresh anything
std::thread:sleep(std::time::Duration::from_secs(100)) // leave us time to check memory in /proc/pid/status
// there, VmPeak / VmSize is around 5 MB in /proc/PID/status
system.refresh_processes()
// there, VmPeak / VmSize is around 1 GB in /proc/PID/status ! 

[GuillaumeGomez]

@EccoTheFlintstone Thanks for the extra information! It matches what I already found out myself:

My wild guess is that it might be coming from rayon/crossbeam (the only memory leak sysinfo has is coming from there). But even though, I'm not sure what would be the "real" origin.

[EccoTheFlintstone]

Hum, indeed it seems to be related to rayon:

https://stackoverflow.com/questions/59205184/how-can-i-change-the-number-of-threads-rayon-uses

fn main() {
rayon::ThreadPoolBuilder::new().num_threads(4).build_global().unwrap();
std::thread:sleep(std::time::Duration::from_secs(100)) 
}

after some tests, it appears rayon consumes about 65 MB / thread (without doing anything in the program yet...) By default, it starts as many thread as there are cores on the machine I dunno if there's something that an be done/configured to lower this memory footprint... (haven't checked on rayon github or doc)

[GuillaumeGomez]

This is surprisingly high. Might be worth investigating what's going on in rayon or check if there is a good alternative.

[Percivalll]

Wow!I have been wondering why there are eight threads running when I execute let mut system = sysinfo::System::new_all();, maybe it is also affected by rayon.

[GuillaumeGomez]

It is voluntary: I update all processes at once this way.

[Percivalll]

Yes,I used let mut system = sysinfo::System::new(); instead, the situation returned to normal.

[GuillaumeGomez]

System::new creates mostly empty content and doesn't call System::refresh_processes, this is why. :)

[EccoTheFlintstone]

Ok, i thin I got an explanation : this is due to Libc

https://www.faircom.com/insights/take-control-of-linux-core-file-size https://ewirch.github.io/2013/11/linux-process-memory-layout.html

foreach thread, libc creates a new heap with 64 MB

I tried compiling the example code for musl (cargo build --target x86_64-unknown_linux-musl --release) and memory consumption is WAY less (67 MB with 32 threads compared to 2 GB with standard libc )

[GuillaumeGomez]

I wonder if it can be changed at runtime? Like that we could try to limit this nonsensical memory usage.

[EccoTheFlintstone]

As for the initial problem (memleak), I take back what I said, I DO get an increase in VmRSS and VmData in /proc/pid/status over time

[EccoTheFlintstone]

I wonder if it can be changed at runtime? Like that we could try to limit this nonsensical memory usage.

You can try using MALLOC_ARENA_MAX env variable but it can have a negative impact on the overall program performances As a matter of fact, every program on linux using libc has this problem

# MALLOC_ARENA_MAX=2 ./your_test_program

workaround:

• use another libc (musl for instance)
• use another memory allocator (jemalloc, tcmalloc, ...)

but this is kind of outside this crate responsibility (on a site note, you could add a note in the doc stating this problem and some of the possible workarounds)

[EccoTheFlintstone]

I get the same behaviour with jemalloc (maybe I missed something), but it seems to be working with tcmalloc (apart from the memleak..) compiling in static with musl is also a know workarount

[EccoTheFlintstone]

Regardless, concerning rayon, you should maybe use a custom ThreadPool (not the global rayon one) and add a config option to specify how many threads one want when using the lib (don't override the rayon global config as people should be able to modify this globally in their application)

[GuillaumeGomez]

It's really not great. :-/ I'd prefer for users to have literally nothing else to do than import the crate. Maybe there is simply another way to get process info without killing performances if we remove the threads... async/await is out of question unfortunately (the /proc files are always ready but when you read them, the kernel generates them so you're kinda stuck). sigh

[Percivalll]

Currently, I have replaced glibc with musl-libc in the entire project. Consider adding a hint to the user in Rustdoc: describe the problem, and prompt the user to use musl to compile and build if they have high performance requirements, which can greatly alleviate this problem.

[GuillaumeGomez]

Since you found out the issue, do you want to open a PR for it? :)

[Percivalll]

Yes, I am pleasure to help users understand this problem and give a more detailed solution. I will open a PR within this week.

[GuillaumeGomez]

Thanks!

[EccoTheFlintstone]

Did you find the root cause of the memleak? There are AFAIK 2 problems here :

• huge memory consumption due to libc threads (in rayon)
• potential memleak if you create a System object in a loop (I DID encounter the problem, just didn't look properly first time I had a look at it)

[GuillaumeGomez]

huge memory consumption due to libc threads (in rayon)

It seems to be the thread overhead and I guess rayon adds its own data. It should be investigated but I think the best is to bring up this issue directly in their repository.

potential memleak if you create a System object in a loop (I DID encounter the problem, just didn't look properly first time I had a look at it)

You should never do that. System has been created to be created once and for all. System statistics are based on comparison with previous values, so it seems very unlikely that you'll get what you expect by doing so and you'll have very bad performance.

[Percivalll]

You should never do that. System has been created to be created once and for all. System statistics are based on comparison with previous values, so it seems very unlikely that you'll get what you expect by doing so and you'll have very bad performance.

But suppose the following situation：

fn func1()->SomeInfomation{
    System::new();
    //Related operations
}
fn func2(){
    let info=func1();
    //Related operations
}

And in this case, func2() will be called frequently (such as regular calls or frequent triggers by other conditions), then there is still a risk of memory leaks. Consider setting System as a global variable or exporting System variable to the scope outside func2()? I don't think it is a good idea (the former will bring additional maintenance overhead and the latter will make the code tedious and difficult to understand).

[Percivalll]

Did you find the root cause of the memleak? There are AFAIK 2 problems here :

• huge memory consumption due to libc threads (in rayon)
• potential memleak if you create a System object in a loop (I DID encounter the problem, just didn't look properly first time I had a look at it)

In the current situation, I prefer to believe the first situation.