`cargo test` eventually fills up `/tmp` and fails

[jbencin]

Describe the bug If you run cargo test repeatedly, /tmp fills up and you start getting errors like this:

thread 'util_lib::bloom::test::test_bloom_counter_is_invertible' panicked at stackslib/src/util_lib/bloom.rs:735:21:
called `Result::unwrap()` on an `Err` value: SqliteFailure(Error { code: DiskFull, extended_code: 13 }, Some("database or disk is full"))

Steps To Reproduce Run cargo test or cargo nextest run repeatedly

Expected behavior Tests should clean up after themselves, or re-use data from previous tests

Environment (please complete the following information):

• OS: Manjaro Linux
• Rust version: 1.78
• Version: 4afa9ef71

[wileyj]

do you have a partition for /tmp of a specific size?

[jbencin]

My /tmp directory is a 16GB tmpfs filesystem, which lives in RAM/swap. That's the default setup for Arch Linux, and probably other distros as well

Even if it doesn't run out of space, it's still consuming a lot of RAM unnecessarily. I have 32GB here, and old files from cargo test can eat up half that

I know most of our devs use MacBooks, and I have no idea how OS X handles this and if it's a problem for them

[jcnelson]

FWIW, neither Debian nor Alpine use a RAM filesystem for /tmp. They use tmpfs, or nothing at all (it's just part of root).

[jbencin]

tmpfs is not ramdisk but uses "virtual memory", which means data can be in either RAM or swap space. See tmpfs kernel docs here. So any distro that mounts tmpfs on /tmp is going to run into problems unless it is rebooted frequently

Am I the only one having an issue with this?

[jcnelson]

tmpfs is not ramdisk but uses "virtual memory", which means data can be in either RAM or swap space.

Right; you could conceivably deal with this problem by increasing your swap space. These distros' /tmp is backed not exclusively by RAM, but by a (user-chosen) amount of disk.

Am I the only one having an issue with this?

Maybe? My /tmp on my Linux boxen are all simply directories.

$ mount | grep 'on /tmp' | wc -l
0

The oldest file in this computer's /tmp is from August 19, 2021.

[jbencin]

Maybe? My /tmp on my Linux boxen are all simply directories.

You're using Debian, right? Debian by default doesn't use tmpfs. Arch does. Not sure about other popular distros like Ubuntu, Fedora, etc.

Even if others aren't getting to the point where they are seeing actual test failures, I still think we should do something about it, because taking up extra RAM or disk space is never a good thing

Proposed Solutions

Used Fixed Names

Instead of using random file/directory names for each invocation of cargo test, use fixed names or psuedo-random names generated from a fixed seed. If data already exists from a previous test, remove it first

Implement Drop trait

Implement a simple wrapper around std::fs::File and implement Drop for it, so that the files are removed when the variable which owns them goes out of scope. Maybe not the right solution if we need to examine files after a failed test

[jcnelson]

Even if others aren't getting to the point where they are seeing actual test failures, I still think we should do something about it, because taking up extra RAM or disk space is never a good thing

We deliberately keep things on disk so we can inspect them on failed test runs, so that's probably not going to change. The value it provides for debugging is immense.

Why can't you just alter your /etc/fstab to not mount /tmp as tmpfs?

[jbencin]

We deliberately keep things on disk so we can inspect them on failed test runs, so that's probably not going to change. The value it provides for debugging is immense.

I understand this. Is it useful to keep all of the data indefinitely, of can we only keep data from the most recent cargo test run?

[wileyj]

i think you have a few options here, but the easiest solutions are platform specific. you may increase that tmpfs size if you want to keep more test data, mount a new partition backed by disk, or delete older temp folders (possibly via makefile). there is also another option to keep the data out of your tmpfs filesystem: have you tried setting TMPDIR env var before running the tests?

ultimately this seems platform specific, so I'm not sure it should be addressed with a change to the repo (possibly a note in the contributing file that this may occur).

On Mon, May 13, 2024 at 12:44 Jeff Bencin @.***> wrote:

We deliberately keep things on disk so we can inspect them on failed test runs, so that's probably not going to change. The value it provides for debugging is immense.

I understand this. Is it useful to keep all of the data indefinitely, of can we only keep data from the most recent cargo test run?

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-core/issues/4779#issuecomment-2108668216, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHGDMAG6NHB7BWPU7R3ZCEJ2DAVCNFSM6AAAAABHRIEZYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBYGY3DQMRRGY . You are receiving this because you commented.Message ID: @.***>