HAFS forecast job restart capability

Description

Provide a clear and concise description of the requested feature/capability. From NCO SPA: Please add this HAFS forecast restart capability at next upgrade.

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.

Status (optional)

Do you (or a colleague) plan to work on adding this feature? @BijuThomas-NOAA, @BinLiu-NOAA, and others will start working on this.

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

The atmosphere-only forecast restart capability has been developed and established with the upcoming HAFSv2 upgrade.

When the forecast job is requested/triggered/instructed to run, the restarted forecast can automatically pick up the latest restart forecast hour (with complete set restart files) and continue finishing the rest of the forecast hours. Currently, from EMC development side, we have tested the writing out restart files for potential restart hours at both every 12 hours and every 24 hours. Both configurations worked fine from our end.

Here some notes/information when considering at what interval we want to write out the restart files for potential restart runs. Currently, HAFS forecast job with operational computation resources can complete 12-hour forecast in ~9 min, and 24-hour forecast in ~18 min. And the size of every set of the forecast restart files is ~ 20GB.

Notes:

• The capability of using the write grid component (dedicated IO groups) to write out the restart files was recently developed/established by EMC/EIB. Without this critical capability, the forecast job would be slowed down by ~2 mins for every set of restart files writing.
• The HAFS application workflow's support for the forecast job restart capability was developed and enabled by the EMC hurricane project team. Developers tested and confirmed the capability through HAFS Dev side Rocoto workflow.
• The HAFSv2 EMC/EIB EE2 Reviewer also tested and confirmed the forecast restart capability working fine with the Ecflow workflow.