General: rustc uses too much memory when compiling librustc

[nielx]

Memory usage of rustc compiling librustc goes through the roof. It is the main reason we cannot build on x86 anymore, and it requires at least 5.2 GB of RAM in x86_64. We need to investigate whether there is something about the Haiku port that causes this abnormality.

[Mark-Simulacrum]

I would suspect that this is not a Haiku issue; you can try turning codegen-units off (set them to 1) within config.toml for the rust build itself; I'm not sure you can configure this behavior for rustc in the wild. It may be worth filing an upstream bug; it's likely we can do better at managing memory usage during codegen (if that's where the issue is).

[nielx]

@Mark-Simulacrum I will try to do some more research on memory usage, to see where exactly memory is allocated. A reason why I think there is a Haiku component associated with it, is that when I watch memory usage on Haiku it just increases, never decreases, while on Windows I do see more variable memory usage during compilation. It may be inefficiency in Haiku's memory allocator.

[nielx]

Two notes:

• It might actually not be rustc to blame (even though it is by no means light on memory usage). Instead, it is the build process that seems to increasingly use more memory as it goes through the steps, indicating that there is some sort of leak. An extra indicator to this is that when the build ends due to being out of memory, restarting it will push it a bit further.
• @waddlesplash has indicated it might be that we are running into the (known) memory leak of fork(). This is used by the Rust standard library to spawn processes, which is something done by cargo. The suggestion is to try to see if we can switch to posix_spawn() instead, which in most cases does not use fork on Haiku. It is not entirely clear if this would fix it though.

The next step is to get support for posix_spawn in libc.

[nielx]

Future changes: /library/std/src/sys/unix/process/process_unix.rs