Upgrade CMake to most current version possible for Trilinos 14.0

[bartlettroscoe]

CC: @trilinos/framework, @keitat, @trilinos/kokkos, @trilinos/kokkos-kernels

Child Issues:

• #10574

Associated Internal Issues:

• SEPW-520: Get CMake 3.22 installed on all Trilinos platforms
• TRILINOSHD-184: Coordinate upgrading minimum version of CMake from 3.17 to 3.22

Description

The TriBITS refactoring work and efforts to reduce the size of TriBITS to use raw CMake would be helped if TriBITS and Trilinos could upgrade the minimum required version of CMake from 3.17 to something much newer. Ideally, we would upgrade to CMake 3.22 (released in November 18, 2021).

A sampling of the new CMake features we could exploit in TriBITS and Trilinos include:

• Use new imported targets generated by FindHDF5.cmake not available in CMake 3.17 to simplify and robustify the Trilinos FindTPLHDF5.cmake module (see below). (CMake 3.20)
• cmake_langauge() command: Better handle lists and function forwarding and allow for function pointers to simplify TriBITS implementation. (CMake 3.18)
• Set ctest test Labels and Details fields at runtime (CMake 3.22). (Allows custom details about failed tests.)
• file(CHMOD_RECURSE …) to replace Linux/bash-specific commands used to fix directory permissions on SNL systems used in Trilinos installation testing and SPARC and EMPIRE + Trilinos Integration testing.
• Support for string(JSON) for native JSON processing to allow more robust handling of data in input files (CMake 3.19)
• Improved HIP support may benefit Kokkos and Trilinos efforts on Frountier (CMake 3.21 for improved HIP support)
• Expanded support for generator expressions in several CMake commands (e.g. TARGET keyword with file(GENERATE …) for CMake 3.19 and add_custom_command() and add_custom_target() for CMake 3.20)
• CMake 3.24.0 adds the --fresh option that robustly removes the CMakeCache.txt file and the CMakeFiles/ directory. So no more having to run rm -rf CMake* before you run cmake ...!

There may be other desirable features discovered in updated CMake versions that would be of use as well.

Note that the last upgrade of the minimum CMake version for Trilinos (and TriBITS) was to 3.17 and was made back on January 28, 2021 in commit b30d8085329d5145af27265222e540458c8cae31. CMake 3.17.0 was tagged on March 20, 2020 which was just 10 months before Trilinos upgraded to CMake 3.17.

Therefore, having Trilinos upgrade to CMake 3.22 in say the next 2 months (by the end of May 2022 before the Trilinos 14.0 release) would mean that Trilinos would be upgrading to CMake 2.22 more than 5 months after it was released. Given the previous upgrade to CMake 3.17 with a 8 month lag from its release, this seems not too unreasonable.

NOTE: If this drags on too long, then we should consider upgrading to CMake 3.23 since it has the added benefits:

• Adds support for IMPORTED_NO_SYSTEM target property (which would allow us to set Trilinos_IMPORTED_NO_SYSTEM_DEFAULT to ON for Trilinos 14.0 and maintain nearly perfect backward compatibility).
• Adds support for micro-seconds %f to string(TIMESTAMP <output_variable> [<format_string>] [UTC]) (allows us to replace expensive and non-portable date calls in TriBITS to get CMake code timing data, see timer_get_raw_seconds()).

Tasks:

• [x] Inquire about getting SEMS to install the module sems-cmake/3.22.z on all SEMS-supported platforms (see SEHELPD-3074) ... sems-cmake/3.22.4 and sems-cmake/3.23.1 are installed (see below)
• [x] Inquire about getting CMake 3.22 installed in ASC DevOps CDE modules
• [x] Inquire about getting CMake 3.22 installed in SPARC modules
• [x] Inquire about getting CMake 3.22 installed in LLNL testbeds ... cmake/3.22.4 and cmake/3.23.1 have been installed on all LLNL testbed machines (see below)
• [x] Inquire about getting CMake 3.22 installed on Vanguard platforms (e.g. 'stria') ... cmake/3.22.3 is installed on 'stria' (see below)
• [x] Verify that CMake 3.22 is installed on all critical platforms
• [x] Post and test MR to upgrade Trilinos PR and 'master' promotion builds to CMake 3.22+ ... See https://cee-gitlab.sandia.gov/trilinos-project/srn-ini-files/-/merge_requests/20
• [x] Wait for merge of https://cee-gitlab.sandia.gov/trilinos-project/srn-ini-files/-/merge_requests/20 (@e10harvey)
• [x] Verify that upgraded CMake not breaking any PR builds (see below)
• [x] Send out announcement to Trilinos stakeholders on raising minimum CMake version from 3.17 to 3.22 (see below)
• [ ] Upgrade all ATDM Trilinos builds to CMake 3.22+
• [ ] Verify that upgraded CMake not breaking any ATDM Trilinos builds (not worse than they already are anyway)
• [ ] Change minimum required version of CMake in TriBITS and Trilinos to CMake 3.22

[rppawlo]

The last time we chose a version, we picked the highest version supported across internal sandia and external hpc machines. Looking at the sems modules, sierra modules, asc cde modules, sparc modules, and the llnl testbeds - none of them have cmake 3.22 or higher. 3.19 looks to be on most of them though. Not saying we should not do this (I would like do this), but it will be more work than last time for all app teams. Will add this to the next product leaders meeting. We can reach out to the app teams and see if they are willing to upgrade.

[bartlettroscoe]

The last time we chose a version, we picked the highest version supported across internal sandia and external hpc machines.

@rppawlo, yes, it will be a prerequisite to get the chosen CMake version installed on all the machines first. We should start pushing that now.

We really need to get the HPC and CSE communities to get into the habit of deploying the newest CMake releases as they come out. It is trivial to install an updated CMake version on any system so we just need to get everyone in the habit of doing so.

CC: @ibaned

[bartlettroscoe]

NOTE: We have a strong motivator to get at least one of the Trilinos PR builds to use at least CMake 3.21 (see #10267). I will see if SEMS can add cmake 3.22 to the set of standard installed modules (see here).

[bartlettroscoe]

FYI: To drive the details of getting CMake 3.22 installed everywhere, I created the internal issue SEPW-520. I will just report back here when it gets done.

NOTE: If this drags on too long, then I will upgrade this request for CMake 3.22 to CMake 3.23 since it has the added benefits:

• Adds support for IMPORTED_NO_SYSTEM target property (which would allow us to set Trilinos_IMPORTED_NO_SYSTEM_DEFAULT to ON for Trilinos 14.0 and maintain nearly perfect backward compatibility.
• Add support for micro-seconds %f to string(TIMESTAMP <output_variable> [<format_string>] [UTC])

[bartlettroscoe]

NOTE: CMake 3.23.0 has just been released as of 6 hours ago:

• https://github.com/Kitware/CMake/releases/tag/v3.23.0
• https://cmake.org/cmake/help/v3.23/

But I think we need to wait at least a month before we upgrade because I am sure that there will be issues found that will be addressed in patch releases 3.23.1, 3.23.2, etc.

[bartlettroscoe]

INVALID (see below)

---

I think we have hit a brick wall with CMake 3.22 (and 3.23 for that matter). The big impediment is that the Spack CMake package does not allow the install of arbitrary versions of CMake downloaded from the main CMake GitHub repo https://github.com/kitware/cmake and instead only supports fixed versions of CMake. And the latest Spack release 0.17.1 (based off of the Spack branch releases/v0.17 created before11/05/2021) only supports CMake 3.21. And the next Spack release 0.18.0 is not slated to come out for some time judging from Spack 0.18.0 release project board and the gab between Spack 0.16.0 and 0.17.0.

I am inquiring if we can get the next patch release Spack 0.17.2 to support CMake 3.22, but that seems to be against CMake policy. See:

• https://github.com/spack/spack/pull/29784
• https://github.com/spack/spack/discussions/29783

More details are given below.

Detailed analysis on Spack CMake versions and releases (click to expand)

If you look at the tip of the Spack ‘develop’ branch you will see it is up to CMake 3.22 at:

• https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/cmake
• https://github.com/spack/spack/commit/6d1da528f77c13108d541c8e8232d8c523f96c92

It looks like they are almost done with the 0.17.2 release from looking at:

• https://github.com/spack/spack/projects/49

(just 3 “in progress” issues left with 25 issues complete).

But sadly, it looks like Spack 0.17.2 may not contain the update for CMake 3.22 from looking at:

• https://github.com/spack/spack/pull/28280
• https://github.com/spack/spack/tree/backports/v0.17.2/var/spack/repos/builtin/packages/cmake

which has only the CMake 3.21 commit:

• https://github.com/spack/spack/commit/c13f915735636a8285538b458d888742a3d27f4d

from October.

Hopefully I can get them to add the CMake 3.22 commit so this will show up in Spack 0.17.2. I created:

• https://github.com/spack/spack/pull/29784

to accomplish that. But it seems that may be against Spack release policy. I posted:

• https://github.com/spack/spack/discussions/29783

about this.

---

[bartlettroscoe]

But since the current Spack release 0.17.1 supports CMake 3.21, that should a slam dunk for every deployment process that uses Spack.

[haampie]

There are no backports of packages in Spack (neither trilinos nor cmake). The develop branch usually has the latest cmake a few days after release.

[bartlettroscoe]

@haampie and @alalazo, see:

• https://github.com/spack/spack/discussions/29783#discussioncomment-2469712

That would fix the CMake deployment problem with all future Spack releases 0.18+.

[bartlettroscoe]

The develop branch usually has the latest cmake a few days after release.

@haampie, as I mentioned in:

• https://github.com/spack/spack/discussions/29783#discussioncomment-2469712

the DevOps teams that are installing CMake on these various systems are using named releases of Spack, not the Spack develop branch.

What we need is a mechanism to allow these Spack packages like CMake and Ninja to be able to download and deploy versions of these packages that come out after the base Spack release version vX.Y.0 is tagged. I outlined a mechanism for doing that.

[haampie]

@bartlettroscoe please stop pinging me all over the place on Github, once is enough. Spack core and the builtin Spack repo are currently tied together. Seems like you want a stable Spack core, but rolling releases for packages. Currently that's simply not supported.

[bartlettroscoe]

@bartlettroscoe please stop pinging me all over the place on Github, once is enough. Spack core and the builtin Spack repo are currently tied together. Seems like you want a stable Spack core, but rolling releases for packages. Currently that's simply not supported.

FYI: To avoid @mentioning and subscribing hammpie again to this Issue, I responded in:

• https://github.com/spack/spack/discussions/29783#discussioncomment-2469953

[bartlettroscoe]

I talked with an internal DevOps team yesterday and they mentioned one path forward is to copy the Spack package directory for CMake spack/var/spack/repos/builtin/packages/cmake/ file (along with its package.py file), put it into a custom Spack "Package Repository", and then make the changes that we need in that copy. That would override the default spack/var/spack/repos/builtin/packages/cmake/ directory and use that for CMake installs.

That is not a great long-term solution but it is something that could allow DevOps teams to install CMake 3.22+ on various platforms using the current Spack v0.17.1 (and other v0.17.z patch releases).

[haampie]

They can also do spack install [email protected] on older spack versions

[bartlettroscoe]

They can also do spack install [email protected] on older spack versions

@haampie, I did not realize that was supported. I will give that a try. If this works, that solves the problem.

[bartlettroscoe]

FYI: I tested out running spack install --no-checksum [email protected] with Spack v0.17.1 (CMake 3.23.0 just got released 2 days ago) and it worked (details below).

So this would seem to be a viable solution for teams using Spack 0.17.1 to install newer versions of CMake.

The only issue that I see is that you have to use the spack install --no-checksum argument to avoid the y/N challenge (passing in -y does not work). Therefore, to use this safely, I think you need to install your main packages using checksumed binaries first using:

$ spack install <full-speck>

and then follow that up with individual calls to:

spack install --no-checksum [email protected]

Details of running spack install [email protected] using Spack 0.17.1 (click to expand)

On my SNL RHEL7 machine 'crf450' (which is not a CEE RHEL7 machine), I ran:

$ spack external find
==> The following specs have been detected on this system and added to /ascldap/users/rabartl/.spack/packages.yaml
[email protected]    [email protected]   [email protected]     [email protected]       [email protected]   [email protected]  [email protected]             [email protected]        [email protected]    [email protected]
[email protected]  [email protected]   [email protected]     [email protected]        [email protected]    [email protected]  [email protected]_272-b10  [email protected]  [email protected]     [email protected]
[email protected]      [email protected]  [email protected]         [email protected]         [email protected]    [email protected]      [email protected]          [email protected]       [email protected]      texlive@20130530
[email protected]      [email protected]     [email protected]  [email protected]  [email protected]  [email protected]   [email protected]    [email protected]       [email protected]  [email protected]

$ spack install -y [email protected]~openssl
==> Bootstrapping clingo from pre-built binaries
==> Warning: Missing a source id for [email protected]
==> Installing pkgconf-1.8.0-2zg3j5kkgcfescjiyrigrdhga7zks4bw
==> No binary for pkgconf-1.8.0-2zg3j5kkgcfescjiyrigrdhga7zks4bw found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ef/ef9c7e61822b7cb8356e6e9e1dca58d9556f3200d78acab35e4347e9d4c2bbaf.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> pkgconf: Executing phase: 'build'
==> pkgconf: Executing phase: 'install'
==> pkgconf: Successfully installed pkgconf-1.8.0-2zg3j5kkgcfescjiyrigrdhga7zks4bw
  Fetch: 0.21s.  Build: 7.08s.  Total: 7.29s.
[+] /home/rabartl/Spack.base/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/pkgconf-1.8.0-2zg3j5kkgcfescjiyrigrdhga7zks4bw
[+] /usr (external openssl-1.0.2k-fips-mjjwdaachzev6hwh27yzbfb4lczuxxqk)
==> Installing ncurses-6.2-c5fr4gw5k7a3jcbkmpknzd4npswnb6pz
==> No binary for ncurses-6.2-c5fr4gw5k7a3jcbkmpknzd4npswnb6pz found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/30/30306e0c76e0f9f1f0de987cf1c82a5c21e1ce6568b9227f7da5b71cbea86c9d.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> ncurses: Successfully installed ncurses-6.2-c5fr4gw5k7a3jcbkmpknzd4npswnb6pz
  Fetch: 0.44s.  Build: 1m 3.56s.  Total: 1m 4.01s.
[+] /home/rabartl/Spack.base/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/ncurses-6.2-c5fr4gw5k7a3jcbkmpknzd4npswnb6pz
==> Installing cmake-3.23.0-se23sydc6krjpdoxka2l76n3nmxdr7et
==> No binary for cmake-3.23.0-se23sydc6krjpdoxka2l76n3nmxdr7et found: installing from source
==> Fetching https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
==> cmake: Executing phase: 'build'
==> cmake: Executing phase: 'install'
==> Warning: Module file /home/rabartl/Spack.base/spack/share/spack/modules/linux-rhel7-haswell/cmake-3.23.0-gcc-4.8.5-se23syd exists and will not be overwritten
==> cmake: Successfully installed cmake-3.23.0-se23sydc6krjpdoxka2l76n3nmxdr7et
  Fetch: 3.86s.  Build: 2m 34.36s.  Total: 2m 38.22s.
[+] /home/rabartl/Spack.base/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/cmake-3.23.0-se23sydc6krjpdoxka2l76n3nmxdr7et

And I tested this CMake 3.23.0 install out with TriBITS and it seems to work.

[bartlettroscoe]

NOTE: An install of CMake is less than 70M using spack install --no-checksum [email protected]. Therefore, having teams install and keep many different installs of CMake on a machine does not take up a lot of disk space. Even 20 installs of CMake at 70M is only 1.4G of space on the machine.

[bartlettroscoe]

I wanted to post some thoughts that I had after spending many hours trying to get CMake 3.22 installed on all of these systems and communicating with lots of people (see the subtasks of the internal issue SEPW-520).

---

We need to have this conversation about CMake at a higher level because the only way that we can maximally leverage forward momentum in CMake feature development and bugfixes (which are substantial as new systems and compiler versions are constantly coming out) is to get all of the facilities and other systems supporting HPC and CSE customers to adjust their deployment process to make installs of new versions of CMake cheap, easy, and regular. CMake is unlike any other piece of software that I can think of in this respect. (And forcing packages to use older versions of CMake creates a lot of added waste needing to work-around issues that are resolved in newer versions of CMake.) Streamlining the CMake deployment process will completely eliminate individual requests for installing updated versions of CMake and allow projects to use whatever newer version of CMake they want/need.

The other option is to move way from the model of relying on facilities to install CMake on the system globally and instead just make it super easy for individual projects and users to install their own CMake instances on any system. Spack seems to be overkill with too many side-effects and complexities to impose on most users who just want CMake and Ninja (which can be built with just a C and C++ compiler and no other upstream dependencies).

NOTE: My current impression is that the latest version of Spack is very attractive for teams that maintain the installs and upgrades of a large number of dependent packages on a given system. In these cases, the time and investment to learn to use Spack is well worth the effort. But that is not the use case I am referring to in the above paragraph. I am just talking about installing latest versions of small executable tools packages with essentially no dependencies like CMake and Ninja (and here the real need is for updated versions of CMake, not Ninja so much).

[alalazo]

Spack seems to be overkill with too many side-effects and complexities to impose on most users who just want CMake and Ninja

In some user directory, use this spack.yaml:

spack:
  specs:
  - cmake
  - ninja
  concretization: together
  config:
    concretizer: clingo
    install_tree:
      root: ./store
  view: ./view

then:

$ ls
spack.yaml
$ spack -e . concretize -f
[ ... ]
$ spack -e . install
[ ... ]

At the end cmake and ninja should be usable by adding ${PWD}/view/bin to the PATH. To update the software, after an update of Spack:

$ spack -e . concretize -f
$ spack -e . install

[bartlettroscoe]

FYI: I just made a request for a package-by-package no-checksum variant for Spack that would allow teams to run a single spack install command and turn of checksum checks for just a newer package for which you trust the source of the tags in spack/spack#29837 to allow:

$ spack install [a lot of specs] [email protected]+no-checksum

where the cmake/package.py file does not have a checksum for CMake version 3.23.

[bartlettroscoe]

@alalazo, is there anyway to run Spack to do that just on the command-line with something like:

$ spack external find
$ spack --install-base <some-base-dir> --no-checksum [email protected] [email protected]

and have it install these under <some-base-dir> as:

<some-base-dir>/
    cmake-3.23.0-<hash>/
        bin/
        ...
    ninja-1.10.2-<hash>/
        bin/
        ...

with no dependencies on the Spack repo that you are using?

That is, you should be able to delete spack repo (and the ~/.spack/ directory and every other file that Spack writes) and have installs be usable. I am even okay with the long hashes at the end of the directory names. People can live with that or rename them if they like.

If we can get Spack to be used in that way, then that solves the problem as far as I am concerned.

[bartlettroscoe]

FYI: CMake 3.22.4 and CMake 3.23.1 has been installed both on the SEMS modules (see internal issue SEPW-522) as the modules sems-cmake/3.22.4 and sems-cmake/3.23.1 and on the LLNL systems (see internal issue SEPW-526) as cmake/3.22.4 and cmake/3.23.1. I was able to test the SEMS modules with TriBITS but I don't have access to the LLNL machines.

@rppawlo, could you or someone with access to the LLNL machines confirm these modules cmake/3.22.4 and cmake/3.23.1 exist and work?

These are all being installed with Spack so I am sure they work (and it is hard to mess up a CMake install, unless you turn on and use OpenSSNL support).

[bartlettroscoe]

Also, note that the cmake/3.22.3 module has been installed on the SNL Vanguard systems since the end of March (see internal issue SEPW-527).

All that is left is to install CMake 3.22 with the CEE CDE and with SPARC. (My guess is that SPARC and install an updated version of CMake very quickly when needed.)

[cgcgcg]

@bartlettroscoe On LLNL's Lassen, the highest version of CMake that I see is 3.21.1

[bartlettroscoe]

Apparently the cmake/3.22.4 and cmake/3.23.1 modules are missing on the LLNL ATS-2 support machines as well.

But at least CMake 3.21 is better than CMake 3.17 which is where we are right now if that is what we have to settle for.

[bartlettroscoe]

On LLNL's Lassen, the highest version of CMake that I see is 3.21.1

@cgcgcg, I am told they are there. Can you please try again, this time with module avail cmake or module spider cmake? Or just try module load cmake/3.22.4 and see if it works?

[cgcgcg]

Just checked again, and now I see cmake/3.23.1.

[bartlettroscoe]

It has been confirmed by @rppawlo and @cgcgcg that the modules cmake/3.22.4 and cmake/3.23.1 have been installed on the needed LLNL testbed machines.

[bartlettroscoe]

I think we are just about there having all of the systems supporting CMake 3.22+ as described in:

• https://sems-atlassian-son.sandia.gov/jira/browse/SEPW-520

Just the Sierra project has not confirmed they have upgraded yet. But since they use Spack and Spack supports checksummed CMake 3.22.4 and 3.32.1, this should be an easy addition.

I think we should pull the trigger on the upgrade of the minimum version of CMake needed by TriBITS and Trilinos to 3.22 any day now.

[bartlettroscoe]

I need to upgrade the minimum version of CMake so that I can properly rename and deprecate some TriBITS functions as per https://github.com/TriBITSPub/TriBITS/pull/493#discussion_r921395885.

I will to get #10614 merged and then I will upgrade the minimum CMake version for TriBITS 'master' to CMake 3.22. Then to get future upgrades of TriBITS to Trilinos, the PR testing of Trilinos will need to be upgraded to CMake 3.22+.