llvm-project icon indicating copy to clipboard operation
llvm-project copied to clipboard
verified publisher icon RadeonOpenCompute

Parallel compilation with Comgr

Open lamb-j opened this issue 2 years ago • 2 comments

Currently Comgr includes mutex locks around compilation actions to avoid unexpected LLVM multi-threading side effects.

https://github.com/RadeonOpenCompute/ROCm-CompilerSupport/blob/amd-stg-open/lib/comgr/src/comgr.cpp#L1252

It would be nice to fix the issues with LLVM preventing multiple threads from maintaining independent LLVM instances. We could then remove these locks in Comgr to allow for parallel compilation

lamb-j avatar Mar 27 '23 21:03 lamb-j

Just to note that the performance is critical for MIOpen (the performance is what MIOpen delivers to its users). Therefore we are going to implement another workaround in the library (leverage forking) in order to use COMgr in truly parallel mode (but this is not even started yet). In other words, the importance of this ticket is high for us.

CC @averinevg

atamazov avatar Mar 28 '23 15:03 atamazov

@lamb-j Hi Jakob, do you really think that this feature is not https://github.com/ROCm/llvm-project/labels/enhancement of Comgr? I am asking because the removal of mutex locks (when LLVM issues are fixed) looks like an enhancement that enables true parallel compilation.

atamazov avatar Feb 16 '24 20:02 atamazov