CUDA support to quantum-espresso package

[bellenlau]

Hello,

this is the extension of the quantum-espresso package to support CUDA.

Thank you,

Laura

[danielecesarini]

@ye-luo this PR is a work of Laura and mine, please revise this PR for us.

@bellenlau is a colleague of mine in CINECA and she works on QE. She will be in charge also of CMake build system for QE (not only of spack package). I will help Laura to address future issues on QE but she will be the primary contact for QE issues on SPACK package and CMake build system.

[bellenlau]

Hello @alalazo, we noticed that the warning message in f9d464d652964cfd73bcaee33bf0b093e38c9e70

# Warning for CUDA without OpenMP
@run_before("cmake")
def caveats(self):
    if "+cuda~openmp" in self.spec:
        tty.warn("CUDA without OpenMP could incur in severe performance degradation !!!")

does not pop out before cmake stage, despite variants match +cuda~openmp.
Any suggestion on this issue?

[danielecesarini]

Hello @alalazo, we noticed that the warning message in f9d464d

# Warning for CUDA without OpenMP
@run_before("cmake")
def caveats(self):
    if "+cuda~openmp" in self.spec:
        tty.warn("CUDA without OpenMP could incur in severe performance degradation !!!")

does not pop out before cmake stage, despite variants match +cuda~openmp. Any suggestion on this issue?

Actually, the tty.warn() prints on the standard output, which is redirected on the build file of SPACK at this time. We would wonder if there is some hook to print messages to warn the user about the configuration choice (that could not be the optimal one). We would do even in the concretization phase not only in the build phase.

[ye-luo]

Right now the blocking issue is that we cannot set openmp variant default value depending on cuda on/off. Implementing a warning is cumbersome and almost invisible. So my suggestions is setting openmp on by default. Users should not stuck in MPI-only land and if they really need MPI-only, OMP_NUM_THREADS=1 can be set. I generally think enable OpenMP by default is a positive thing.

[ye-luo]

Right now the blocking issue is that we cannot set openmp variant default value depending on cuda on/off. Implementing a warning is cumbersome and almost invisible. So my suggestions is setting openmp on by default. Users should not stuck in MPI-only land and if they really need MPI-only, OMP_NUM_THREADS=1 can be set. I generally think enable OpenMP by default is a positive thing.

@danielecesarini @bellenlau what do you think? Let us finalize this PR.

[bellenlau]

Right now the blocking issue is that we cannot set openmp variant default value depending on cuda on/off. Implementing a warning is cumbersome and almost invisible. So my suggestions is setting openmp on by default. Users should not stuck in MPI-only land and if they really need MPI-only, OMP_NUM_THREADS=1 can be set. I generally think enable OpenMP by default is a positive thing.

@danielecesarini @bellenlau what do you think? Let us finalize this PR.

OpenMP as a default always is fine - even if it is not the same when using cmake only . I will push asap.

[bellenlau]

@amd-toolchain-support apparently the pipeline to build QuantumEspresso fails when configuring amdfftw with OpenMP (https://gitlab.spack.io/spack/spack/-/jobs/3003806). Could you provide further insight on this?

Here is how the configuration fails:

checking for dummy main to link with Fortran 77 libraries... unknown
configure: WARNING: *** Couldn't figure out how to link C and Fortran; using default Fortran wrappers.
checking for OpenMP flag of C compiler... unknown
configure: error: don't know how to enable OpenMP

[alalazo]

@bellenlau Probably you already saw the message, but in case you missed it, you can reproduce the failure locally following these instructions:

To reproduce this build locally, run:
    spack ci reproduce-build https://gitlab.spack.io/api/v4/projects/2/jobs/3003806/artifacts [--working-dir <dir>]
If this project does not have public pipelines, you will need to first:
    export GITLAB_PRIVATE_TOKEN=<generated_token>
... then follow the printed instructions.

printed at the end of the logs.

[danielecesarini]

@bellenlau Probably you already saw the message, but in case you missed it, you can reproduce the failure locally following these instructions:

To reproduce this build locally, run:
    spack ci reproduce-build https://gitlab.spack.io/api/v4/projects/2/jobs/3003806/artifacts [--working-dir <dir>]
If this project does not have public pipelines, you will need to first:
    export GITLAB_PRIVATE_TOKEN=<generated_token>
... then follow the printed instructions.

printed at the end of the logs.

@alalazo we think that this is a bug of Spack (or about the CI), because the problem is related to the AMDFFW library on a container that runs on ARM/Graviton2 instance (we don't depend on AMDFFTW but only to the virtual provider FFTW-API@3). But this is look very weird, could you check the pipeline?

[alalazo]

@danielecesarini This is not a bug in Spack, nor in the CI. The changes you make here, and the overuse of conflicts, lead clingo to add penalties to certain providers. The result is that with the changes in this PR the "optimal" configuration is when amdfftw is in the DAG[^1].

For complicated packages, like quantum-espresso, my experience is that directives that "tell Spack what to do" work better than directives that tell Spack what to do not. Can you check if this diff:

diff --git a/var/spack/repos/builtin/packages/quantum-espresso/package.py b/var/spack/repos/builtin/packages/quantum-espresso/package.py
index ff8efb6345..1e1007c775 100644
--- a/var/spack/repos/builtin/packages/quantum-espresso/package.py
+++ b/var/spack/repos/builtin/packages/quantum-espresso/package.py
@@ -68,10 +68,9 @@ class QuantumEspresso(CMakePackage):
     # Need OpenMP threaded FFTW and BLAS libraries when configured
     # with OpenMP support
     with when("+openmp"):
-        conflicts("^fftw~openmp")
-        conflicts("^amdfftw~openmp")
-        conflicts("^openblas threads=none")
-        conflicts("^openblas threads=pthreads")
+        depends_on("fftw+openmp", when="^fftw")
+        depends_on("amdfftw+openmp", when="^amdfftw")
+        depends_on("openblas threads=openmp", when="^openblas")
 
     # Add cuda support
     with when("%nvhpc"):

works for you? With that I was able to make quantum-espresso concretize by default using fftw and openblas [^1]: obviously amdfftw doesn't build neatly, so maybe it's worth pinging @amd-toolchain-support and @cgmb about that

[cgmb]

1. obviously amdfftw doesn't build neatly, so maybe it's worth pinging @amd-toolchain-support and @cgmb about that

The AMD CPU and GPU math library teams are entirely separate. I'm a part of the GPU group, so I'm afraid I don't have any contacts with the amdfftw developers. I'm an occasional user of the library, but I know basically nothing about autotools. AOCL is the domain of AMD Toolchain Support.

[bellenlau]

@alalazo Thank you for the hint! Changes are included in commit 4cb6bc4ee2fc64ce717675f843b88f1228e2bc5b.

For complicated packages, like quantum-espresso, my experience is that directives that "tell Spack what to do" work better than directives that tell Spack what to do not. Can you check if this diff:

diff --git a/var/spack/repos/builtin/packages/quantum-espresso/package.py b/var/spack/repos/builtin/packages/quantum-espresso/package.py
index ff8efb6345..1e1007c775 100644
--- a/var/spack/repos/builtin/packages/quantum-espresso/package.py
+++ b/var/spack/repos/builtin/packages/quantum-espresso/package.py
@@ -68,10 +68,9 @@ class QuantumEspresso(CMakePackage):
     # Need OpenMP threaded FFTW and BLAS libraries when configured
     # with OpenMP support
     with when("+openmp"):
-        conflicts("^fftw~openmp")
-        conflicts("^amdfftw~openmp")
-        conflicts("^openblas threads=none")
-        conflicts("^openblas threads=pthreads")
+        depends_on("fftw+openmp", when="^fftw")
+        depends_on("amdfftw+openmp", when="^amdfftw")
+        depends_on("openblas threads=openmp", when="^openblas")
 
     # Add cuda support
     with when("%nvhpc"):

works for you? With that I was able to make quantum-espresso concretize by default using fftw and openblas

We further added depends_on("amdblis threads=openmp", when="^amdblis") to improve the default concretization of amd blas.

[danielecesarini]

@ye-luo I think that we are ready to finalize this PR.

[ye-luo]

@ye-luo I think that we are ready to finalize this PR.

I will take a look on Friday. @bellenlau could you rebase all the commits on top of the develop?

[bellenlau]

@ye-luo comments have been updated in ff4d4c672fdf47bab3e3547a302157a3c103912a

[alalazo]

Thanks!

[bellenlau]

@alalazo Apparently, merging is blocked because of a lack of permit. Do you have any hints on this issue?

[alalazo]

@bellenlau We are reworking CI on GitHub, and had issues running pipelines for a couple of days. That part is solved now.