Linux makefile build serializes during middle of build

[gigadude]

The linux build serializes down to a single core during the middle of the build (I'd estimate 90% of the build time is spent in this serialized section). The start and end of the build use all cores as expected. I can't see any dependencies that would be causing this, but I'm no cmake expert.

[gpx1000]

Try this: cmake --build . -- -j 7

in other words, you want to pass to Make that it should build in parallel.

[gigadude]

I tried -j 20 (I've got 16 cores), also tried omitting it and using MAKEFLAGS=-j16, there is parallelism during the start and end of the build (KSysGuard shows all cores 100%) but the middle section is just using a single core, and only 6% at that.

[TomAtkinsonArm]

Hey @gigadude,

If you take a GraphViz export from the CMake it shows that all third parties are linked against the framework. This means that once all third parties are build in parallel the build system more than likely is parsing the framework in a serialized manner. I may be wrong... If you could provide any log dumps or information that you think is valuable to identifying the bottleneck that would be great.

This is poor design, but I would like to highlight that Vulkan Samples is a developer education resource first and the priority here is the samples themselves and not having the most efficient build system. One method to improve the build system would be to split the frameworks components behind interfaces (The best designs are those based from behavior and not a strict application flow). This will allow the framework to be compiled as multiple static libraries which are only linked with third parties which they use. Everything can then be compiled in parallel and linked at the end appropriately.

The difficult aspect here is not the identification of the problem, its clear as day, but more how should we go about solving the problem. Most developers on this project are using their spare time to contribute, review and maintain code and so such a large change can be incredibly hard and slow to introduce to the project.

I am open to more solutions and my comments here may be out of place and not the root cause of the problem, I wanted to highlight how large I think the change is and the road blocks that may stop it in the short term. Please feel free to contribute a proposal or an implementation which helps fix this issue

[TomAtkinsonArm]

Hi @gigadude,

It was decided on the Vulkan Samples Working Group call today to investigate this issue further as other users are also experiencing slower builds.

If possible, can you provide a build log? This way we can highlight where in the build you are experiencing this serialization

[gpx1000]

Here's a profile pass of a build log.txt

Posting it here for reference for everyone. I haven't gone through it yet.

[gary-sweet]

This is becoming a pain point for me when trying to test new samples. I have a 96 core build server and I'm building with -j.

The first 40% of the build takes about 30 seconds. The remainder takes a further 7 minutes.

[TomAtkinsonArm]

Looking through the actions. Before Plugins we built in around 25m. After plugins this is now 35m. The biggest change here is compiling samples as static libraries instead of object libraries

You may need to download the above picture to zoom and see it. Here we can see a normal graph of dependencies. The only issue is its circular. Everything links against framework and then everything links against apps which links against framework... CMake is magic but probably not magic enough to resolve this design

[gpx1000]

I ran into that problem here's how I got past it:

What matters in "circular dependencies" is the linking phase. CMake correctly says that's bad practice when trying to make non-static libs (i.e. objects that are self contained like executables or shared libs). The path towards a simple fix is to use the INTERFACE headers defined on the target like so: target_include_directories( foo PUBLIC $<TARGET_PROPERTY:bar,INTERFACE_INCLUDE_DIRECTORIES>) set_target_properties(foo PROPERTIES INTERFACE_LINK_LIBRARIES $<TARGET_PROPERTY:bar,INTERFACE_LINK_LIBRARIES)

That way you can maintain the header dependencies. As well as any consuming application will capture the dependencies as originally envisioned.

However, a better fix is what I'm currently working on. I hate circular dependencies.... straightening out the path correctly is simpler and easier for people consuming CMake to not duplicate bad practices.