Memory usage is high, resulting in Out of memory OS kill

[azriel91]

trafficstars

When running cargo outdated on a repository with many small crates (46), this error appears after a while:

cargo outdated
[1]    5557 killed     cargo outdated

The system is Ubuntu 16.04 with 8 GB physical memory. Investigating with:

dmesg | less

gives:

[22541.218190] Out of memory: Kill process 5557 (cargo-outdated) score 611 or sacrifice child
[22541.218196] Killed process 5557 (cargo-outdated) total-vm:8599764kB, anon-rss:5518604kB, file-rss:4kB, shmem-rss:0kB
[22541.529600] oom_reaper: reaped process 5557 (cargo-outdated), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

I haven't checked code, but if it's trying to compile the whole repo in /tmp which is mounted on my RAM, perhaps that explains it.

Can see from this graph that the memory usage (green) spikes before it's killed:

A proposed fix is to take in a parameter for the temporary location to compile. Maybe there's some parts of the code that can be memory optimized as well.

[0xpr03]

Just wanted to note that it works without such problems here on 35 root dependencies (at 340 dependencies to build). With max of 119MB heap, taking 1 second.

[azriel91]

Btw, are you using a workspace, where the root crate is non-virtual (aka has source code)?

edit: With version 0.8.0, in my workspace that has 60 member crates, on my Windows box with 16 gigs of memory, it just hangs.

ran for 3 hours, using 13 gb memory before I cancelled it

Interrupted it to show how long it was running:

[16:32:52] autex_master master
$ time cargo outdated -w

real    179m8.593s
user    0m0.015s
sys     0m0.141s

[19:32:14] autex_master master

with `--depth 8` specified it ends in 9 minutes, but says everything is up to date

$ time cargo outdated -p will --depth 8
All dependencies are up to date, yay!

real    9m0.528s
user    0m0.000s
sys     0m0.062s

[0xpr03]

No I'm not using any workspace, but I'm using one path-local crate. That could be the difference. Edit: Also I'm on a linux box, not sure about how exactly this could influence this, but for example a huge amount of file IO could make it a difference.

[0xpr03]

Oh and it could also be some recursion in your dependencies, if the depth is making a difference.

[deg4uss3r]

cargo-outdated does build a similar project in temp, is this still happening with 0.9.10?

If so can you provide some more information:

• Which OS and Version are you running where this is still happening?
• Which rustc version are you using?
• A Cargo.toml for me to test?

Thanks!

[khuey]

I have a (not open source) workspace with a total of 766 transitive dependencies and cargo outdated eventually consumes all my RAM and dies. This is on Linux, rustc 1.44, and cargo-outdated v0.9.11.

[TriplEight]

I would like to have this to be fixed and willing to help. Still learning Rust so I'd appreciate mentoring if possible. I have a large workspace project that reproduces it. @deg4uss3r are you still fancy?

[deg4uss3r]

Hi @TriplEight thanks so much for the ping on this, but as you can tell from my late reply, I'm not the best person to help at this time 😅

I'd ask @kbknapp and see if he or anyone he's been working with recently can assist.

I do plan on trying to get back in here but I have a cross-country move coming up (again) and need some personal time.

[kbknapp]

I'd be interested to see if this is still the case on the latest (v0.13.1). If anyone has a repo that can reproduce this I'd like to see.

[khuey]

I have a (not open source) workspace with a total of 766 transitive dependencies and cargo outdated eventually consumes all my RAM and dies. This is on Linux, rustc 1.44, and cargo-outdated v0.9.11.

It still happens on the workspace mentioned here with rustc 1.70 and cargo-outdated 0.13.1

[kbknapp]

I haven't tested this yet, but assuming the root cause is that we're using a temp dir that in some instances is a tempfs mount and exhausting RAM; a simple solution would be to give users control over where the temp dir is located via something like a --tempdir flag or similar.

[khuey]

I ran it on a machine with 256GB of RAM and it went for 40 minutes and OOMed there too.

[kbknapp]

That sounds like a recursion issue. Do you have any circular dev or path dependencies? Does it still happen when you use a max depth similar to another poster here?

[khuey]

It does go away when I constrain max depth enough. I don't really understand how recursion is possible though, aren't dependency graphs inherently acyclic?

[kbknapp]

Dev dependencies are allowed to be cyclic in Rust.

That helps though, I'll do some testing on it this week and report back.

[starwolfy]

Performing cargo oudated on my Rust project results in RAM usage climbing over 32GB , crashing my Ubuntu desktop. I cannot use this tool for my project.

[kbknapp]

I'm sorry that happened 😞 But thank you for providing that project as this really helps and I was able to reproduce the unbounded memory growth which makes me thing it's some cyclic dependency.

[kbknapp]

Looks like at least in @starwolfy's case it's hitting an infinite loop at elaborate_workspace.rs:232.