rocBLAS [Issue]: Build error with different cpp files

Problem Description

Build error with different cpp files.

port gfx1103, gfx1150, gfx1151 based on https://src.fedoraproject.org/rpms/rocblas/blob/74df24057a4579f507a50431aaa96ae7484d1567/f/0001-add-gfx1103-support-for-rocBLAS.patch to rocblase

use python rmake.py -a "gfx1103;gfx1150;gfx1151" -t G:\AMD\Asus\Tensile --no_hipblaslt

Note that if specified gfx1103 or gfx1150 or gfx1151 , it will build successfully But for multiple architecture, it encounter error with different CPP. there is no detail logs

Clang version is following clang version 19.0.0git ([email protected]:Compute-Mirrors/llvm-project b3dbdf4f03718d63a3292f784216fddb3e73d521)

Operating System

10.0.22631

CPU

AMD Ryzen 7 7700X 8-Core Processor

GPU

AMD Radeon RX 7900 XTX

ROCm Version

ROCm 6.1.0

ROCm Component

hipBLAS

Steps to Reproduce

Adding patch for gfx1103, gfx1150, gfx1151. build with following command in windows python rmake.py -a "gfx1103;gfx1150;gfx1151" -t G:\AMD\Asus\Tensile --no_hipblaslt

1st time failed [49/481] Building CXX object library/src/CMakeFiles/rocblas.dir/blas2/rocblas_spr2.cpp.obj FAILED: library/src/CMakeFiles/rocblas.dir/blas2/rocblas_spr2.cpp.obj C:\opt\rocm\6.1\bin\clang++.exe -DBUILD_WITH_TENSILE -DROCBLAS_BETA_FEATURES_API -DROCBLAS_INTERNAL_API -DROCBLAS_TENSILE_LAZY_LOAD=1 -DROCBLAS_TENSILE_SEPARATE_ARCH=1 -DROCM_USE_FLOAT16 -DTENSILE_DEFAULT_SERIALIZATION -DTENSILE_MSGPACK=1 -DTENSILE_USE_HIP -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__=1 -Drocblas_EXPORTS -IG:/AMD/Asus/rocBLAS/library/src -IG:/AMD/Asus/rocBLAS/library/include -IG:/AMD/Asus/rocBLAS/library/include/internal -IG:/AMD/Asus/rocBLAS/library/src/include -IG:/AMD/Asus/rocBLAS/build/release/include/rocblas/internal -IG:/AMD/Asus/rocBLAS/build/release/include/rocblas -IG:/AMD/Asus/rocBLAS/build/release/include -IG:/AMD/Asus/rocBLAS/library/src/blas3/Tensile -IG:/AMD/Asus/rocBLAS/build/release/virtualenv/Lib/site-packages/Tensile/Source/lib/include -isystem C:/opt/rocm/6.1/include -DWIN32 -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_CRT_SECURE_NO_WARNINGS -D_SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING -Wno-ignored-attributes -DHIP_CLANG_HCC_COMPAT_MODE=1 -fms-extensions -fms-compatibility -D__HIP_ROCclr__=1 -D__HIP_PLATFORM_AMD__=1 -D__HIP_HCC_COMPAT_MODE__=1 -O3 -DNDEBUG -std=c++17 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -fvisibility-inlines-hidden --offload-compress -Wno-unused-result -Werror=vla -mllvm=-amdgpu-early-inline-all=true -mllvm=-amdgpu-function-calls=false -x hip --offload-arch=gfx1103 --offload-arch=gfx1150 --offload-arch=gfx1151 -parallel-jobs=3 -Wno-format-nonliteral -Wno-deprecated-declarations -Wno-ignored-attributes -Wdll-attribute-on-redeclaration -fdelayed-template-parsing -MD -MT library/src/CMakeFiles/rocblas.dir/blas2/rocblas_spr2.cpp.obj -MF library\src\CMakeFiles\rocblas.dir\blas2\rocblas_spr2.cpp.obj.d -o library/src/CMakeFiles/rocblas.dir/blas2/rocblas_spr2.cpp.obj -c G:/AMD/Asus/rocBLAS/library/src/blas2/rocblas_spr2.cpp error: error opening 'library\src\CMakeFiles\rocblas.dir\blas2\rocblas_spr2.cpp.obj.d': permission denied 1 error generated when compiling for gfx1151. [53/481] Building CXX object library/src/CMakeFiles/rocblas.dir/blas2/rocblas_spr_strided_batched.cpp.obj ninja: build stopped: subcommand failed. Traceback (most recent call last): File "G:\AMD\Asus\rocBLAS\rmake.py", line 582, in main() File "G:\AMD\Asus\rocBLAS\rmake.py", line 575, in main if run_cmd(exe, opts): ^^^^^^^^^^^^^^^^^^ File "G:\AMD\Asus\rocBLAS\rmake.py", line 538, in run_cmd proc = subprocess.run(program, check=True, stderr=subprocess.STDOUT, shell=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2032.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'ninja.exe -j 5 all' returned non-zero exit status 1.

2nd time build failed log [279/481] Building CXX object library/src/CMakeFiles/rocblas.dir/src64/blas3/rocblas_syrkx_herkx_kernels_64.cpp.obj FAILED: library/src/CMakeFiles/rocblas.dir/src64/blas3/rocblas_syrkx_herkx_kernels_64.cpp.obj C:\opt\rocm\6.1\bin\clang++.exe -DBUILD_WITH_TENSILE -DROCBLAS_BETA_FEATURES_API -DROCBLAS_INTERNAL_API -DROCBLAS_TENSILE_LAZY_LOAD=1 -DROCBLAS_TENSILE_SEPARATE_ARCH=1 -DROCM_USE_FLOAT16 -DTENSILE_DEFAULT_SERIALIZATION -DTENSILE_MSGPACK=1 -DTENSILE_USE_HIP -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__=1 -Drocblas_EXPORTS -DROCBLAS_INTERNAL_ILP64 -IG:/AMD/Asus/rocBLAS/library/src -IG:/AMD/Asus/rocBLAS/library/include -IG:/AMD/Asus/rocBLAS/library/include/internal -IG:/AMD/Asus/rocBLAS/library/src/include -IG:/AMD/Asus/rocBLAS/build/release/include/rocblas/internal -IG:/AMD/Asus/rocBLAS/build/release/include/rocblas -IG:/AMD/Asus/rocBLAS/build/release/include -IG:/AMD/Asus/rocBLAS/library/src/blas3/Tensile -IG:/AMD/Asus/rocBLAS/build/release/virtualenv/Lib/site-packages/Tensile/Source/lib/include -isystem C:/opt/rocm/6.1/include -DWIN32 -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_CRT_SECURE_NO_WARNINGS -D_SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING -Wno-ignored-attributes -DHIP_CLANG_HCC_COMPAT_MODE=1 -fms-extensions -fms-compatibility -D__HIP_ROCclr__=1 -D__HIP_PLATFORM_AMD__=1 -D__HIP_HCC_COMPAT_MODE__=1 -O3 -DNDEBUG -std=c++17 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -fvisibility-inlines-hidden --offload-compress -Wno-unused-result -Werror=vla -mllvm=-amdgpu-early-inline-all=true -mllvm=-amdgpu-function-calls=false -x hip --offload-arch=gfx1103 --offload-arch=gfx1150 --offload-arch=gfx1151 -parallel-jobs=3 -Wno-format-nonliteral -Wno-deprecated-declarations -Wno-ignored-attributes -Wdll-attribute-on-redeclaration -fdelayed-template-parsing -MD -MT library/src/CMakeFiles/rocblas.dir/src64/blas3/rocblas_syrkx_herkx_kernels_64.cpp.obj -MF library\src\CMakeFiles\rocblas.dir\src64\blas3\rocblas_syrkx_herkx_kernels_64.cpp.obj.d -o library/src/CMakeFiles/rocblas.dir/src64/blas3/rocblas_syrkx_herkx_kernels_64.cpp.obj -c G:/AMD/Asus/rocBLAS/library/src/src64/blas3/rocblas_syrkx_herkx_kernels_64.cpp PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script. Exception Code: 0xC0000005 Exception Code: 0xC0000005 Exception Code: 0xC0000005 [283/481] Building CXX object library/src/CMakeFiles/rocblas.dir/blas2/rocblas_trsv_kernels.cpp.obj ninja: build stopped: subcommand failed. Traceback (most recent call last): File "G:\AMD\Asus\rocBLAS\rmake.py", line 582, in main() File "G:\AMD\Asus\rocBLAS\rmake.py", line 575, in main if run_cmd(exe, opts): ^^^^^^^^^^^^^^^^^^ File "G:\AMD\Asus\rocBLAS\rmake.py", line 538, in run_cmd proc = subprocess.run(program, check=True, stderr=subprocess.STDOUT, shell=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2032.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'ninja.exe -j 5 all' returned non-zero exit status 1.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

I use windows

Additional Information

None

Oct 04 '24 08:10 Jay19751103

The dependency failure looks like a known compiler issue with multi-threading. The rocblas patch in develop is just to disable parallelism for a single compilation unit, see commit in develop branch 759485ca7c63a26365b7bdc4fa3b5b128f277fd1. It randomly occurs when more than one gfx arch was listed. The 6.1 branch has different code, but the gist of it is -parallel-jobs=3 needs to be removed from the compiler flags so if in your env var HIPCC_COMPILE_FLAGS_APPEND then remove it there.

Oct 08 '24 17:10 TorreZuk

After vcpkg reinstall packages, it does not have this problem now. Close the issue.

Nov 29 '24 01:11 Jay19751103