hpx
hpx copied to clipboard
STEllAR-GROUP
With the new CUDA folder structure, CuBlas is not found
Expected Behavior
Octo-Tiger should compile as before
Actual Behavior
it seems that cublas_v2.h is not found, even if the header is in the path.
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/hydro_cuda_interface.cpp:1:9: warning: #pragma once in main file
1 | #pragma once
| ^~~~
In file included from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/cublas_executor.hpp:24,
from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/modules/async_cuda.hpp:19,
from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/flux_cuda_kernel.cpp:13:
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/custom_blas_api.hpp:39:14: fatal error: cublas_v2.h: No such file or directory
39 | #include <cublas_v2.h>
| ^~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/hydrolib.dir/build.make:132: CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/flux_cuda_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
In file included from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/cublas_executor.hpp:24,
from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/modules/async_cuda.hpp:19,
from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/reconstruct_cuda_kernel.cpp:2:
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/custom_blas_api.hpp:39:14: fatal error: cublas_v2.h: No such file or directory
39 | #include <cublas_v2.h>
| ^~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/hydrolib.dir/build.make:146: CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/reconstruct_cuda_kernel.cpp.o] Error 1
ptxas info : 45 bytes gmem
gmake[2]: Leaving directory '/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/octotiger/build-kokkos'
gmake[1]: *** [CMakeFiles/Makefile2:176: CMakeFiles/hydrolib.dir/all] Error 2
gmake[1]: Leaving directory '/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/octotiger/build-kokkos'
gmake: *** [Makefile:136: all] Error 2
Steps to Reproduce the Problem
... Please be as specific as possible while describing how to reproduce your problem.
- Build HPX with CUDA
- Compile Octo-Tiger
Specifications
... Please describe your environment
- HPX Version: Master
- Platform (compiler, OS): gcc 10.3/11.2 and CUDA 11.4 and cmake 3.19 and 3.21
@diehlpk could you please check whether the HPX preprocessor constant HPX_HAVE_GPUBLAS is defined (should be in hpx/config/defines.hpp)?
@diehlpk could it be that you have to add the CUDA::cublas target to the octotiger executable target? HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.
@hkaiser I think it does, because we include the cublas_v2.h here
https://github.com/STEllAR-GROUP/hpx/blob/ae45453ace32e6f5da0af9b63e09cae0876e3ada/libs/core/async_cuda/include/hpx/async_cuda/custom_blas_api.hpp#L39
@hkaiser In addition, we include hipblas as well
https://github.com/STEllAR-GROUP/hpx/blob/ae45453ace32e6f5da0af9b63e09cae0876e3ada/libs/core/async_cuda/include/hpx/async_cuda/custom_blas_api.hpp#L14
I never build with HIP and I do not know if that will be some issue.
HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.
@hkaiser I would assume that HPX propagates the dependency if HPX includes the header.
HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.
@hkaiser I would assume that HPX propagates the dependency if HPX includes the header.
You're right, my fault. I'll have another look.
cublas/hipblas is optional because hip installations do not necessarily include hipblas (https://github.com/STEllAR-GROUP/hpx/blob/581d3fb174e55eb1a8ba68d54dc9108e3cbac82b/cmake/HPX_SetupCUDA.cmake#L47-L53). @diehlpk can you try a newer version of CMake? CMake has been notoriously bad at finding various CUDA libraries, especially in the newer versions.
cublas/hipblas is optional because hip installations do not necessarily include hipblas (https://github.com/STEllAR-GROUP/hpx/blob/581d3fb174e55eb1a8ba68d54dc9108e3cbac82b/cmake/HPX_SetupCUDA.cmake#L47-L53). @diehlpk can you try a newer version of CMake? CMake has been notoriously bad at finding various CUDA libraries, especially in the newer versions.
@msimberg even with 3.21 it does not work.
@msimberg I think the issue is that the HPX CUDA scripts do find cuBLAS, but this is not propagated to dependent projects.