Parallel build is no longer parallel

[wiredfool]

What did you do?

make debug or make install

What did you expect to happen?

Expect compilation to happen across all the cores, taking ~ 2-3 seconds.

What actually happened?

Single threaded, taking ~15 seconds. There are something like 60 compilation units, and I have at least 8 cores.

What are your OS, Python and Pillow versions?

• OS: Mac + Linux
• Python: 3.12
• Pillow: 11.2, main

Sometime between ~6 years ago and now, probably when the setuptools migrated around or we went to the newer build tool specifications, the parallel option here: https://github.com/python-pillow/Pillow/blob/main/setup.py#L381 stopped working.

I can confirm that the code there is running, self.parallel is set correctly, but setuptools is apparently not using it anymore.

[radarhere]

I'm not convinced this is the case, or at least I don't think it's completely simple.

Looking at setuptools, here's build_extensions() checking self.parallel, and calling _build_extensions_parallel() if it is truthy - https://github.com/pypa/setuptools/blob/3250c25197b299658cfd4d0db67770fc29b47277/setuptools/_distutils/command/build_ext.py#L478-L484

    def build_extensions(self) -> None:
        # First, sanity-check the 'extensions' list
        self.check_extensions_list(self.extensions)
        if self.parallel:
            self._build_extensions_parallel()
        else:
            self._build_extensions_serial()

If I adjust setup.py with

diff --git a/setup.py b/setup.py
index 5d41e27d9..6af34afad 100644
--- a/setup.py
+++ b/setup.py
@@ -955,7 +955,10 @@ class pil_build_ext(build_ext):
         tk_libs = ["psapi"] if sys.platform in ("win32", "cygwin") else []
         self._update_extension("PIL._imagingtk", tk_libs)
 
+        print("Start parallel check. self.parallel =", self.parallel)
+        build_ext._build_extensions_parallel = lambda x: print("Parallel method called")
         build_ext.build_extensions(self)
+        int("Finish check")

then I see that build_ext._build_extensions_parallel is called in both macOS and Linux. So setuptools thinks it is running in parallel.

[radarhere]

I think the real answer is https://github.com/pypa/cibuildwheel/issues/879#issuecomment-944910872

The distutils feature only builds extensions in parallel, not the files of the extension. Which makes it useless for any package that only ships a single extension (which is common)

[wiredfool]

That comment points to pybind11 having a parallel compile plugin, which seems to be a way forward.