Linking is really slow with Ninja CMake generator, but not Unix Makefiles

[chillenb]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.13.1

Problem description

When building a large pybind11 module with GCC+CMake+Ninja, I find that the linking step is serial and takes forever. When the Makefiles generator is specified, linking (LTO) uses all available cpus, and linking is much faster.

environment: Conda-forge, linux x64

GCC 12.3.0 CMake 3.30.0 GNU Make 4.4.1 Ninja 1.12.1

The problem seems related to the fact that Ninja does not support the GNU Make jobserver protocol. Therefore, the -flto flag does serial LTO with Ninja and parallel LTO with Make. You need to explicitly set -flto=auto or -flto=n when you use Ninja for reasonable link times.

It might be a good idea to change set(thin "") to set(thin "=auto") here: https://github.com/pybind/pybind11/blob/bb05e0810b87e74709d9f4c4545f1f57a1b386f5/tools/pybind11Common.cmake#L327 Clang supports -flto=auto too (with Clang, it means the same as -flto).

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression