[wip] sycl branch

[oksanaguba]

[oksanaguba]

Older states of this branch contain more caar/remap/sph tests that do not require mpi. See commit 206d0caad8df02fe903c00e4f8af454ed658f538 (before prepare to merge to master part1 in which i started cleaning some of that). There were 3 small tests, remap (file r1.cpp), sphere op, probably, divergence (file s1_ml), and caar Pre Exchange (c1_ut), and file tester_simple.cpp. The tests (macro cxx_unit_test_simple) were created to investigate performance from machine to machine for small parts of code and without mpi. They did not use F counterpart (cause the needed perf flags), but used compute_error script for bfb or error evaluations. Also see TESTER_NOMPI pragma, which was needed to compile without mpi. Also CaarFunctorImpl had infrastructure to brake it into smaller parts.

I could see that having these tests around (at least, caar) is probably useful in the future, but organizing them properly should be a separate PR, for which i do not have time.

[oksanaguba]

@bartgol or @ambrad in this branch i revived possibility to use external kokkos build instead of pointing ekat to a new kokkos version. i still prefer to have such option, but do you object to it?

[ambrad]

@oksanaguba is that for standalone Homme? I suspect that's an easier case for supporting an external Kokkos build than the CIME case. In any case, it's fine with me; I believe the option was lost by accident.

[bartgol]

@oksanaguba You can tell cmake to use a prebuilt Kokkos, even with the current infrastructure. You should be able to do it via either an env or a CMake variable. E.g., via env var:

export Kokkos_ROOT=/home/lbertag/workdir/libs/ekat/ekat-install/branch

cmake <options> /path/to/source

For me this gives the output:

-- Looking for a Kokkos installation ...
-- Looking for a Kokkos installation ... FOUND
--   Kokkos_DIR: /home/lbertag/workdir/libs/ekat/ekat-install/branch/lib64/cmake/Kokkos

The same should be achievable by setting the CMake var Kokkos_ROOT (or it may be Kokkos_DIR in that case, I'm not sure). Either way, you always point to the install root folder, of course.

[oksanaguba]

Ready for review, please, @bartgol and @ambrad (i will run tests meanwhile).

[ambrad]

Two points for consideration:

1. I agree with Luca that it's not clear the bool->int conversion to work around the system issue is, in its current form, as minimal as it can be. It might be worth thinking about whether the workaround can be implemented with substantially fewer modifications.
2. This PR has a lot of micro commits, 103. Ideally, these would be squashed to a few meaningful ones.

[oksanaguba]

Thanks, Luca and Andrew. Luca used lazygit and made a new branch with a few commits out of this one. If you do not like int-bool changes, we can just wait till the issue resolves itself, and then redo this work. If possible, I would not want to spend more time cleaning that change (but i will check HV functor if i had an unnecessary change there).

[oksanaguba]

see #6594 .