linking a large number of obj files exceeding system mmap limit

[feizhengfeng]

I was trying to use mold to link a large number of obj files. mold fails with mmap error when loading ~65k files. Looks like linux system's default vm.max_map_count (65530) was reached. What's the best way to workaround it?

I tried "ld -r" to combine multiple obj files and reduce the number. It worked but there seemed to be some dwarf debug section issue.

[rui314]

Obviously the easiest way to fix your issue is to increase max_map_count. If you can't do that, you may want to combine object files into a few .a files using the ar command to reduce the number of input files.

What was the issue with ld -r? That's orthogonal to this resource limit issue, but if there's an issue, we need to fix it.

[Alcaro]

If you can't do that, you may want to combine object files into a few .a files using the ar command to reduce the number of input files.

Note that this changes the output binary if an object file contains only unreferenced symbols; that object file will be omitted. If one of said symbols is a global variable's constructor, the change may affect the program's behavior; I've seen a few bugs caused by exactly that.

I suspect that's why OP tried ld -r. It's less risky.

[rui314]

Note that this changes the output binary if an object file contains only unreferenced symbols; that object file will be omitted. If one of said symbols is a global variable's constructor, the change may affect the program's behavior; I've seen a few bugs caused by exactly that.

Correct. But you can wrap an archive file with --whole-archive and --no-whole-archive. E.g. -Wl,--whole-archive foo.a -Wl,--no-whole-archive.

[feizhengfeng]

Obviously the easiest way to fix your issue is to increase max_map_count. If you can't do that, you may want to combine object files into a few .a files using the ar command to reduce the number of input files.

What was the issue with ld -r? That's orthogonal to this resource limit issue, but if there's an issue, we need to fix it.

a.c:
int main() {
    return 0;
}

$ gcc -c -g a.c
$ mold -r a.o -o a2.o
$ gcc a2.o
ld: a2.o: bad reloc symbol index (0x9 >= 0x8) for offset 0x6 in section `.debug_aranges'
ld: a2.o: error adding symbols: bad value
collect2: error: ld returned 1 exit status

$ gcc -B<path to mold dir> a2.o
collect2: fatal error: ld terminated with signal 11 [Segmentation fault]
compilation terminated.

Above is using recent gcc/ld. With old gcc/ld, it fails even without '-g'.

[rui314]

The -r issue should be resolved with the above change.

[feizhengfeng]

Thanks. I'll use the ar workaround, and try ld -r later.