Error: mold-1.6.0 - Too many open files

[Cerberios]

Hello there! I got error while compile MESA driver with mold. Environment - Gentoo

mold: fatal: cannot open src/gallium/drivers/nouveau/libnouveau.a.p/nvc0_nvc0_video_vp.c.o: Too many open files clang-15: error: linker command failed with exit code 1 (use -v to see invocation)

Please, instruct me what kind of information need provide additionally in order to help devs fix it. Thanks!

[rui314]

Are you running it on a VM, in a Docker, or on a special filesystem?

[Cerberios]

I use Laptop PC with real HDD with disk model: ST1000LM048-2E71 Separated partition for /root with LUKS encryption and EXT4 filesystem. clang version 15.0.4 I tried to switch from stable 1.6.0 version to git-based version of mold - the same error. mold 1.6.0 (bd555b33abad4abe23e755a2c1e0f773672c997f; compatible with GNU ld) ---My flags for compiler--- SPECFLAGS="-O3 -march=native -mtune=native -pipe -fopenmp -mllvm -polly -mllvm -polly-parallel -mllvm -polly-vectorizer=polly -mllvm -polly-omp-backend=LLVM -mllvm -polly-num-threads=4 -mllvm -polly-scheduling=dynamic -mllvm -polly-scheduling-chunksize=1 -mllvm -polly-ast-use-context -mllvm -polly-invariant-load-hoisting -mllvm -polly-loopfusion-greedy -mllvm -polly-run-inliner -mllvm -polly-run-dce -fno-math-errno -fno-trapping-math -falign-functions=32 -fno-semantic-interposition -flto=thin"

CFLAGS="${SPECFLAGS}" CXXFLAGS="${SPECFLAGS}"

LDFLAGS="${LDFLAGS} -Wl,--hash-style=gnu -Wl,-O1,--sort-common,-z,relro,-z,now -Wl,--enable-new-dtags -Wl,-O3,-Bsymbolic-functions,--as-needed -Wl,--thinlto-jobs=all -lomp -fuse-ld=mold"

[rui314]

Please run ulimit -a and paste its output.

[Cerberios]

cerberos /tmp # ulimit -a real-time non-blocking time (microseconds, -R) unlimited core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 30 file size (blocks, -f) unlimited pending signals (-i) 30935 max locked memory (kbytes, -l) 8192 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 30935 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

[rui314]

"open files 1024" should be enough for any build. It looks like this is similar to https://github.com/rui314/mold/issues/797. Are you sure your build environment is not peculiar? That's why I asked if you are running on a VM or something. For example, are you running it on Linux on Windows?

[Cerberios]

Mr. Rui, thank You for idea. I running on native hardware PC (Laptop) under Linux Gentoo. Seems I fixes issue with next hack: cerberos /tmp # echo "ulimit -n 64000" >> .bashrc ; echo "ulimit -n 64000" >> .bash_profile After this "fix" compilation with the same environment went OK

[gcarq]

I can reproduce this issue on Gentoo on bare metal with LUKS and BTRFS and an open file limit of 1024.

clang version: 15.0.7 mold version: 1.10.1

CFLAGS="-flto=thin -march=native -O3 -pipe"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-fuse-ld=mold -flto=thin -unwindlib=libunwind -Wl,-O2 -Wl,-z,relro -Wl,--as-needed"

Raising the file limit to 4096 temporarily fixes the issue.

[rui314]

I don't think we can fix it on our side, as I believe LTO uses that many files.

[aperezdc]

I manage to hit this very easily whenever I try to build WebnKit (surprise, it has lots of files!), top the point that I have to use ulimit -n 16384; but both LLD and the BFD linker manage to successfully link WebKit with the default maximum of 1024 open files (ulimit -n 1024). No idea why that's the case, but they are clearly doing something differently than Mold—it may not be possible to fix this due to how Mold works, but I thought you may want to know that the other linkers work with the lower limit.

[thesamesam]

I don't think we can fix it on our side, as I believe LTO uses that many files.

Could mold call prlimit(2)?

[negril]

I don't think we can fix it on our side, as I believe LTO uses that many files.

Could mold call prlimit(2)?

@rui314 could you implement this? This sadly makes mold hardly useable as default linker on gentoo for larger projects with LTO.

[rui314]

If BFD linker manages to do the same without hitting the limit, it could mean that mold leaks for describes in some cases. What is your compiler and compiler version, and what program are you trying to build?

[negril]

Compiler is clang 16.0.6 Program is firefox 115.0.2 mold is 1.11.0

[thesamesam]

Others already gave an example of webkit-gtk and firefox too. In https://github.com/rui314/mold/issues/1026#issuecomment-1512463301, @ishitatsuyuki says this is expected. So, just call setprlimit instead of having users call ulimit.

Also, you said that earlier yourself at https://github.com/rui314/mold/issues/851#issuecomment-1408605193.

[LtdJorge]

Same happens here for big programs, also on Gentoo with Clang ThinLTO. For example, mesa and qemu failed for this reason (1024 fds).