Extremely slow linking with ld, taking several minutes

[ocroquette]

I apologize if this is the wrong place for this report. In this case I would be grateful to be pointer to a more appropriate one.

We have a pretty big C++ project built since several years with the MINGW toolchain on Windows. During the last years, our builds took always longer, and we could pinpoint it last year to the linker needing extremely long times, far above one minute, even 9 minutes in the most extreme case. It impacts in particular the executables containing unit tests, slowing down the local iterations for the project developers. The situation is degrading with time, probably due to the increasing code size.

As an experiment, we built the project with lld from the clang toolchain, and got immediately much smaller link times, typically below one second. Just a handful of executables need a few seconds at most to link.

From my online research, we are not the only one to have performance issues with ld on Windows, but I didn't find any report at this scale. It looks like we hit a pathological case. I cannot share the code base, and since the problem is obviously related to the code size, I cannot provide a minimal example. But I would be ready to provide more information and spend time investigating the issue if it helps, even though we are probably going to switch this project to lld.

[mati865]

I've observed BFD (ld.exe) on Windows to be roughly two times slower than on Linux for very small project (about dozen libs): https://github.com/rust-lang/rust/issues/76992#issuecomment-696424119 This was with with exceptions added to Windows Defender.

I haven't dug deeper whether it's due to NTFS or something else.

[ocroquette]

Thanks for the feedback. In our case, we are talking about factors of several hundreds though.

I have ruled out our anti-virus, since the behavior is the same with Windows Defender.

[mati865]

I have ruled out our anti-virus, since the behavior is the same with Windows Defender.

Windows Defender slows down executables like any other popular AV software.

[ocroquette]

Since I observed the same times with Defender and our corporate AV, I assumed that it was not related to this. To find out for sure, I just measured without any AV (e.g. Windows Defender Real-time protection was off), and ld behaves the same: it took 6 minutes to link a specific executable, where lld needed only 3s.

[MarcelHB]

Has this become any worse recently? I don't know exactly what updates made it happen, but between February '22 and now, ld.exe turned into consuming 5-10 times the time for one project.

[mati865]

There was Binutils upgrade to 2.38 but it shouldn't have made things worse and surely not by this amount. Maybe it's AV software issue?

[MarcelHB]

Unlikely, I only have Windows Defender. I just disabled it, and still counted 80+ seconds for something that was 15s, maybe 20s.

Performance of v12.1 gcc.exe appears unchanged though.

[mati865]

If you downgrade to old version via pacman -U https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-binutils-2.37-5-any.pkg.tar.zst does it help?

[MarcelHB]

Thanks, it's appears to be better by 10% over multiple some test runs, but that's not the magnitude I was hoping for. I also may need to revert the project's revision again to look for any unforeseen side effects of changes.

[ocroquette]

We are using lld instead of ld since more than one year now. Performance issues belong to the past, and we didn't see any side-effect. It is a drop-in replacement. We never looked back and I highly recommend it. You can still use GCC as compiler of course.

[MarcelHB]

Good idea, I just tested that and I'm back at 15s.

[ocroquette]

I am glad to hear that. It would be interesting to know what triggered the issue though, to help locating the root cause, if you have time.

[mati865]

we didn't see any side-effect

One side effect that you can run into is import library format. When you create DLL with LLD it will output short import library that is also used by all non ancient MSVC tools. ld.bfd has problems with linking them but for creating executables there should be no downsides.

[Chairn]

Hello, i also have this problem since i moved from "legacy" mingw and mingw64 where link time is especially long using DDNet's codebase (https://github.com/ddnet/ddnet). Here are the command executed by cmake call for link only and their respective time:

## Client linking (very slow)
$ time C:/MinGW64/bin/ar.exe cr CMakeFiles/DDNet.dir/objects.a @CMakeFiles/DDNet.dir/objects1.rsp

real    0m12,465s
user    0m0,000s
sys     0m0,031s

$ time C:/MinGW64/bin/g++.exe -fdiagnostics-color=always -Werror=class-memaccess -g  -mwindows -Wl,--whole-archive CMakeFiles/DDNet.dir/objects.a -Wl,--no-whole-archive  -o DDNet.exe -Wl,--out-implib,libDDNet.dll.a -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles/DDNet.dir/linklibs.rsp

real    0m32,080s
user    0m0,000s
sys     0m0,031s

## server linking (normal)
$ time C:/MinGW64/bin/ar.exe cr CMakeFiles/DDNet-Server.dir/objects.a @CMakeFiles/DDNet-Server.dir/objects1.rsp

real    0m0,723s
user    0m0,000s
sys     0m0,015s

$ time C:/MinGW64/bin/g++.exe -fdiagnostics-color=always -Werror=class-memaccess -g   -Wl,--whole-archive CMakeFiles/DDNet-Server.dir/objects.a -Wl,--no-whole-archive  -o DDNet-Server.exe -Wl,--out-implib,libDDNet-Server.dll.a -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles/DDNet-Server.dir/linklibs.rsp

real    0m2,804s
user    0m0,000s
sys     0m0,015s

Note that both takes between 2 and 3 seconds on Linux, but the client is especially slow on Windows.

Version used: g++.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0 GNU ld (GNU Binutils) 2.30 Copyright (C) 2018 Free Software Foundation, Inc.

[Biswa96]

@Chairn That specific toolchain is not provided by msys2's mingw packages. You are using this one https://sourceforge.net/projects/mingw-w64/.

[Chairn]

Well, not really better with MSYS2 packages:

## client linking (slow)
$ time /c/msys64/mingw64/bin/ar.exe qc CMakeFiles/DDNet.dir/objects.a @CMakeFiles/DDNet.dir/objects1.rsp                                                                                        
real    0m17,699s
user    0m0,015s
sys     0m0,000s

$ time /c/msys64/mingw64/bin/c++.exe -fdiagnostics-color=always -g -mwindows -Wl,--whole-archive CMakeFiles/DDNet.dir/objects.a -Wl,--no-whole-archive -o DDNet.exe -Wl,--out-implib,libDDNet.dll.a -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles/DDNet.dir/linklibs.rsp
real    0m28,305s
user    0m0,000s
sys     0m0,000s

$ c++ --version
c++.exe (Rev2, Built by MSYS2 project) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ ld --version
GNU ld (GNU Binutils) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.