Isolated start-up and shut-down graphs.

[hjoliver]

[UPDATE: likely to be superseded by a more general solution; partially addresses the retrospectively-written issue #7020]

Supersede #4912 Supersede #5036 (Really addresses the original #4903 but specifically just for start-up and shut-down graphs)

Implements isolated NON-CYCLING startup and shutdown graphs as special cycle points, to avoid the pain of eternal dependence on initial and final tasks.

[UPDATE from below] I initially called the new isolated graphs "alpha" and "omega" to avoid any confusion with the concepts of initial, start, final, stop, cycle points. However, that sounds a bit too quirky and I've switched to "startup graph" and "shutdown graph", which is descriptive, and it shouldn't be confusing if we consistently use "startup" and "shutdown" (not "start" and "stop").

How it works:

• at start-up, the startup graph loads and runs to completion
• then the main (cycling) graph loads, and runs to completion
• then the shutdown graph loads and runs to completion

Also:

• restart works as normal, in any of the graphs
• you can manually trigger tasks in any section, in which case (e.g.) startup/foo will coexist in the task pool with 3/bar etc.
  • if you need to retrigger an initial install task (e.g.) part way through a workflow run, you may or may not want to pause the main flow while it runs - to handle this we should implement flow pause (which is intended anyway, but not done yet)

EXAMPLE

[scheduling]
    [[graph]]
        startup= "a => b"
        shutdown= "x => y"
        R1 = "foo => bar"
[runtime]
    [[a, b, x, y, foo, bar]]

Result:

$ cylc cat-log junk | grep '=> running'
INFO - [startup/a submitted job:01 flows:1] => running
INFO - [startup/b submitted job:01 flows:1] => running
INFO - [1/foo submitted job:01 flows:2] => running
INFO - [1/bar submitted job:01 flows:2] => running
INFO - [shutdown/x submitted job:01 flows:3] => running
INFO - [shutdown/y submitted job:01 flows:3] => running

Check List

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Applied any dependency changes to both setup.cfg and conda-environment.yml.
• [x] Tests are included (or explain why tests are not needed).
• [x] CHANGES.md entry included if this is a change that can affect users
• [x] Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/559
• [x] If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

[hjoliver]

I'm tentatively putting this one back to 8.1.0.

• It's ready to go and shouldn't be hard to review (and the underlying principles were discussed at length and agreed)
• It's a nice feature that many users will like (it makes special behaviour at start much easier to understand and configure)
• It has some nasty conflict potential if left too long

[hjoliver]

Merged master and deconflicted.

[hjoliver]

@dpmatthews - pinging you for a big-picture review (given your involvement in the discussions that led to this).

[hjoliver]

Squashed and rebased.

[hjoliver]

(Rebased)

[hjoliver]

(Rebased)

[hjoliver]

(Rebased)

[dwsutherland]

I'm annoyed I didn't review this one earlier.. A welcome feature for NIWA Ops. (have cheekily added this to the 8.4.0 milestone)

[oliver-sanders]

8.4.0 is intended to be a small release filling in the remaining intervention holes:

• Cylc remove proposal.
• Skip mode proposal.

This is a large low-priority feature that requires some thought.

Cheekily bumping this back until we are able to give it the time it requires.

[hjoliver]

Yes the new intervention stuff is higher priority, but at the least I'd really like to prioritize this soon after 8.4.0.

This has been requested by many users for a long time. It looks "bigger" than it really is, in that it doesn't mess with the nasty bits of scheduler internals. And, we did discuss and agree it several years back.