Fix fs::dir iterator with the `linux_raw` backend can cause memory explosion

[hackersontwohouse]

Description 🐛

When using rustix::fs::Dir using the linux_raw backend, it's possible for the iterator to "get stuck" when an IO error is encountered. Combined with a memory over-allocation issue in rustix::fs::Dir::read_more, this can cause quick and unbounded memory explosion (gigabytes in a few seconds if used on a hot path) and eventually lead to an OOM crash of the application.

Since <Dir as Iterator>::next calls Dir::read, which in turn calls Dir::read_more, this means an IO error encountered during reading a directory can lead to rapid and unbounded growth of memory use.

PoC:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // create a directory, get a FD to it, then unlink the directory but keep the FD
    std::fs::create_dir("tmp_dir")?;
    let dir_fd = rustix::fs::openat(
        rustix::fs::CWD,
        rustix::cstr!("tmp_dir"),
        rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::CLOEXEC,
        rustix::fs::Mode::empty(),
    )?;
    std::fs::remove_dir("tmp_dir")?;

    // iterator gets stuck in infinite loop and memory explodes
    rustix::fs::Dir::read_from(dir_fd)?
        // the iterator keeps returning `Some(Err(_))`, but never halts by returning `None`
        // therefore if the implementation ignores the error (or otherwise continues
        // after seeing the error instead of breaking), the loop will not halt
        .filter_map(|dirent_maybe_error| dirent_maybe_error.ok())
        .for_each(|dirent| {
            // your happy path
            println!("{dirent:?}");
        });

    Ok(())
}

If a program tries to access a directory with its file descriptor after the file has been unlinked (or any other action that leaves the Dir iterator in the stuck state), and the implementation does not break after seeing an error, it can cause a memory explosion. An attacker knowledgeable about the implementation details of a vulnerable target can therefore try to trigger this fault condition via any one or a combination of several available APIs. If successful, the application host will quickly run out of memory, after which the application will likely be terminated by an OOM killer, leading to denial of service.

CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H