global-workflow icon indicating copy to clipboard operation
global-workflow copied to clipboard

Convert staging job to python and yaml

Open KateFriedman-NOAA opened this issue 8 months ago • 28 comments

Description

This PR converts the staging job from shell to python and introduces the use of yaml.

Changes in this PR:

  1. Rename scripts/exglobal_stage_ic.sh to scripts/exglobal_stage_ic.py.
  2. Update jobs/JGLOBAL_STAGE_IC to use .py script extension. Move COM* variable declarations and member loop down into yaml and python respectively. Move GDATE/gPDY/gcyc settings up to JJOB from ex-script and replace with newer cycle variables (as done in forecast job).
  3. Create parm/stage folder to hold newly created stage.yaml.j2, which both mimics forecast-only functionality in existing scripts/exglobal_stage_ic.sh and adds functionality for cycled mode.
  4. Create ush/python/pygfs/task/stage.py to house staging job python functions for call from scripts/exglobal_stage_ic.py.
  5. Remove stage_ic job rocoto dependencies from xml. Do not need and removes area of duplicate maintenance.
  6. Add cycled staging jobs for gdas and enkf suites.
  7. Rename model_data to model for issue #2686

There will now be distinct stage_ic jobs for each RUN: gdasstage_ic, gfsstage_ic, enkfgdasstage_ic, stage_ic (for gefs).

Related work was done to set up new symlink folder structure under supported platform ICSDIR folder for use by updated staging job.

Resolves #2475 Resolves #2650 Resolves #2686

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO

How has this been tested?

Ran all of the CI tests (either just staging jobs or end-to-end) on WCOSS2. Ran some of the CI tests on Hera and Hercules.

Ran some control CI tests from develop and compared outputs against test CI from staging branch. Outputs reproduced between control and test.

KateFriedman-NOAA avatar Jun 03 '24 15:06 KateFriedman-NOAA