EPIC: Initial TriBITS Modernization and Generalization

[bartlettroscoe]

Description

This Epic will plan for and track the execution of the TriBITS modernization effort to address the immediate needs of Trilinos, Kokkos and the ECP Spack ecosystem. The goal is to do this in a way that will provide as much backward compatibility as reasonably possible to avoid having to rewrite thousands of CMakeLists.txt and *.cmake files in Trilinos, Drekar, Charon2, CASL VERA and other projects using TriBITS or breaking external customers using Trilinos as much as is reasonable.

Refactoring plan

• [x] #63: Rename branch ‘63-merge-tpls-packages’ to ‘63-merge-tpls-packages-1’, do some clean-up to improve understanding without changing any behavior (knowing it is incomplete) (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• [x] #63: Branch ‘63-merge-tpls-packages-2' (PR #374) for more cleanup of code building dependency graph.
• [x] #337: Extract whatever info/advice we can get out of the similar VTK Module upgrade to modern CMake.
• [x] #274: Create tool to make all macro, function, and command names and calls lower case and apply to TriBITS 'master' branch. (This will make the future refactorings easier by making the code easier to read.)
• [x] trilinos/Trilinos#8498: Announce and then remove support for Makefile.export.* files and push to Trilinos ‘develop’.
• [x] #63: #377
• [x] #381: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (in support of #63 and #299).
• [x] #363: Have full GitHub Actions testing set up and posting to CDash for all needed use cases.
• [x] #342: Refine and agree to initial targets for the refactoring.
• [x] #299: Refactor the guts of TriBITS to use modern CMake target-based propagation of everything instead of variables. (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• [x] #63: Finish up basics of treating TriBITS package and TPLs as just “packages” and provide a few examples/tests in TriBITS itself (Use Case 3: Configure/build pointing to a subset of already installed TriBITS packages in same repo). (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• [x] Refactor Trilinos (and perhaps Kokkos some) to allow pre-building and installing Kokkos and the pointing to that installed Kokkos (ignoring packages/kokkos) when building the rest of Trilinos. (Merge to Trilinos ‘develop’) (see https://github.com/trilinos/Trilinos/issues/11545)
• [x] Update Spack 'develop' Trilinos package.py file to depend on kokkos version 4.1.00 (4.1.0) when using Trilinos 'develop' (and all future releases of Trilinos starting with 14.4) ... See https://github.com/spack/spack/pull/39397
• [x] Refactor Trilinos (and perhaps Kokkos and KokkosKernels some) to allow pre-building and installing Kokkos and KokkosKernels and the pointing to those installed Kokkos and KokkosKernels when building the rest of Trilinos (see https://github.com/trilinos/Trilinos/issues/12002) ... See https://github.com/kokkos/kokkos-kernels/pull/2020
• [ ] Refactor the independent SEACAS TriBITS project to pull in pre-installed Kokkos and Zoltan packages (will allow separate Kokkos, KokkosKernels, SEACAS and remaining Trilinos packages to co-exist as Spack packages and linked into single downstream packages and APPs).
• [ ] Address other refactorings and cleanups as they are exposed (as funding and time permits) ...

[bartlettroscoe]

CC: @keitat

FYI: The ExaWind project has specifically asked to have Trilinos build against pre-installed Kokkos before SC 2021. Given the plan above, we should have this done by the end of FY21.

[bartlettroscoe]

CC: @keita, @fnrizzi, @marcinwrobel1986, @MikolajZuzek

The merge of the branch '63-merge-tpls-packages-1' to TriBITS 'master' is now complete (see https://github.com/TriBITSPub/TriBITS/issues/63#issuecomment-847906370). I can now continue with some much more aggressive refactorings towards #63 and #299. Next, I will do some more clean up refactorings to improve understanding of the code base to make the next steps easier.

[bartlettroscoe]

FYI: I talked with @gsjaardema about this plan and added the sub-story above:

• Refactor the independent SEACAS TriBITS project to pull in pre-installed Kokkos and Zoltan packages (will allow separate Kokkos, KokkosKernels, SEACAS and remaining Trilinos packages to co-exist as Spack packages and linked into single downstream packages and APPs)

That should be easy once we can build Trilinos against pre-installed Kokkos and KokkosKernels.

[keitat]

xSDK members are looking for a way to import Kokkos' build information. They need optimization flags and compiler (nvcc_wrapper for CUDA build) information.

[bartlettroscoe]

xSDK members are looking for a way to import Kokkos' build information. They need optimization flags and compiler (nvcc_wrapper for CUDA build) information.

@keitat, from Spack? This is not a Spack-related epic. This epic will enable more flexibility with Spack but does not directly address Spack packages.

[bartlettroscoe]

FYI: The move to modern CMake targets for TriBITS and Trilinos is complete. See https://github.com/TriBITSPub/TriBITS/issues/299#issuecomment-1228937911.

[bartlettroscoe]

FYI:

We have a demonstration ready for pre-building/installing Kokkos, then KokkosKernels, then the rest of Trilinos. This demos the building and installing Kokkos and KokkosKernels separately as their own Spack packages and then building the rest of Trilinos as its own Spack package (and similar use cases). (And we can further partition Trilinos into smaller Spack packages in the future as we would like starting with SEACAS.)

All of the critical changes on the TriBITS side should be complete. The question is what to do about the current incompatibilities between the TriBITS builds of Kokkos and KokkosKernels and the native non-TriBITS CMake builds for those packages and what is expected by downstream Trilinos packages. (Changes can be made on either side to accommodate some of the differences.)

[bartlettroscoe]

FYI, with the merge of PR:

• https://github.com/trilinos/Trilinos/pull/11863

Trilinos 'develop' and Kokkos 'develop' now support pre-isntalling Kokkos with the native Kokkos non-TriBITS CMake build system.

[bartlettroscoe]

FYI: Here is the Spack PR that has the Trilinos Spack package using the Kokkos Spack package:

• https://github.com/spack/spack/pull/39397

[bartlettroscoe]

FYI: Looks like the changes to KokkosKernels was made to allow it to be installed using its native non-TriBITS CMake build system. See:

• https://github.com/kokkos/kokkos-kernels/pull/2020

[bartlettroscoe]

I am going to descope to remove the SEACAS work (but I think has been implicitly done by SEACAS developers). Looking at:

• https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/seacas/package.py

it does not looks like the SEACAS Spack package has yet been updated to use external Kokkos or Zoltan, but someone added a ToDo for this a long time ago:

• https://github.com/spack/spack/commit/ccce3b79ab22952becc946d953dab0f501d47afd

I don't think that is actually hard to do at this point, and SEACAS has already been updated for external Pamgen (see here and the commit https://github.com/sandialabs/seacas/commit/2d4c7ce66f417090259b57898c8d9d3cc02dce0d).

[bartlettroscoe]

Closing as complete (which should have been done a long time ago ...)

[bartlettroscoe]

Closing as complete (which should have been done a long time ago ...)