How to prevent a future sub-graph from running

[hjoliver]

We recently kind-of agreed that:

Remove-like functionality is the right way to chop off a bit of graph

However the cylc remove extension proposal does not achieve that except for current active tasks:

• for past tasks (n<0) it erases flow history, primarily to allow easy re-run in the same flow
• for active tasks (n=0) it does chop off the downstream graph
• for future tasks (n>0) it does nothing, and punts the problem to skip mode

[Aside: it's a pity we didn't call it cylc erase]

Skip mode is natural for "skipping over" a bunch of tasks that can easily be identified as a group (e.g. a whole cycle, or a family). But it is not so good for the fairly common use case of preventing an arbitrary future side-graph from running downstream of a particular task. By "arbitrary" I mean, in particular, that I may not have foreseen the need for this and so the entire sub-graph was not configured in advance to be in a family expressly to make use of skip mode easy.

Example, in a multi-model workflow external circumstances dictate that I no longer need to run model-x in the next cycle, and by implication its entire post-processing and product-generation side-graph. The natural way to do this is to simply force-expire next/model-x (expire means: for external reasons we no longer need to run this task).

However, this will cause a future stall if we did not have the foresight to set model-x:expire? as optional.

Here I am deliberately and knowingly chopping off future graph for a good reason, so a future stall is extremely unhelpful. I can't prevent the unwanted stall, I have to wait for it to happen before I can deal with it.

---

[Note this is not a contrived use case - it is exactly like clock-expire in every respect except that the external reason for expiration is not linked to the clock time - and hence potentially not easily identified as an expire use case before starting the workflow.]

---

Proposal

A two-step intervention that makes the potential danger clear to the user:

1. cylc set --out=expire next/model-x
   • scheduler expires the tasks and warns next/model-x did not complete required outputs
   • without further intervention this will cause a future stall, when the flow encounters the task
2. cylc remove --expire next/model-x - prompt: warning: this will cut the graph off at next/model-x, do you really want to do this? - has the same effect as removing the task once the future stall has occurred, so the DB must record the removal rather than simply erase the history (which would cause it to run again when the flow reaches it)