Triggering parentless task in new flow caused it to spawn in later cycles

[MetRonnie]

Reproducible Example

[scheduler]
    allow implicit tasks = True
[scheduling]
    cycling mode = integer
    [[graph]]
        P1 = """
            end[-P1] => start
            start & compile => end
        """

[runtime]
    [[end]]
        script = false

cylc trigger example//1/compile --flow=new

causes 2/compile, 3/compile etc to spawn again.

Expected Behaviour

Only 1/compile should spawn.

Additional Context

Tested and reproduced all the way back in 8.2.6

[hjoliver]

Not a bug!

In any flow, including the original flow and "new" ones, parentless tasks continue on to future cycles by magically spawning next instances, since they have no parents to do it "on demand". If not for this you couldn't trigger a new flow that traverses the whole graph just like the original flow.

We probably do need an option to not do that, which would make this more of a feature request.

We have definitely discussed this before somewhere but I can't find the associated issue.

~At the moment you can prevent the flow on with cylc set --out=expired --flow=2 on the next-cycle parentless task before triggering flow 2 (which means "don't run the task in flow 2 even if it becomes ready"). Then later you'll have to remove the manually-expired task in flow 2~ - see #6221

Correction: for parentless tasks, this only prevents the one future instance (the manually expired one) from running. That's by design. If a parentless task expires or fails (say) that doesn't mean other instances beyond that cycle should not run.

So for the moment, you have to trigger with --flow=none to avoid flow-on to future cycles. Which obviously is no good if you do want flow-on within the same cycle.

[oliver-sanders]

I'm not sure what the question is here, what were you trying to achieve with this trigger?

• Option 1: I wanted to rerun all tasks in the first cycle:
  • With reflow:
    • With reflow, we cannot presently set a "termination point" beyond which the new flow will not continue beyond.
    • So this approach would not have worked in this example because the end[-P1] => start would have caused the new flow to roll on into the next cycle.
    • If we didn't spawn the parentless tasks in future cycles, then the workflow would stall which would also be undesireable.
  • With remove:
    • The remove proposal will allow you to remove all tasks in the cycle, priming them for re-running (but not actually triggering them).
  • With group trigger:
    • The "group trigger" proposal extends remove functionality by also identifying and triggering the start tasks in the group.
• Option 2: I just wanted to re-run this one task:
  • With reflow:
    • Use --flow=none
  • With remove:
    • Remove the old instance and trigger the task.
  • With group trigger:
    • Trigger the task (I think the proposal makes this the default behaviour).
• Option 3: I wanted to re-run the workflow from the start.
  • With reflow:
    • You'll also need to trigger compile i.e cylc trigger example//1/compile --flow=new.
  • With remove:
    • Possible, but icky.
  • With group triggers:
    • Possible, might be icky, depends on the fine details of task matching.

IMO, the observed behaviour is correct for --flow=new. If we didn't spawn the tasks in future cycles, then the workflow would stall as a result of the trigger which would be highly undesirable.

Suggest ensuring that the intended use case is adequately covered by proposed interventions and closing this if it is?

[MetRonnie]

The use case I can think of is "I want to re-run a chain in a particular cycle point, it will catch up with the blocked flow 1 and flow-merge, therefore it is unexpected that the task I triggered starts running in other cycle points"

[oliver-sanders]

The best match for that would be group trigger.

As I understand it (I haven't had the time to go through in detail yet):

cylc trigger workflow//1  # remove all tasks (and their outputs) and re-run from the start

[hjoliver]

I need to reread the above, but I think the "question" is resolved: @MetRonnie reported expected (and necessary) behavior, not a bug. But we should spin off a new feature issue: the ability to optionally trigger parentless tasks without spawning new cycles - and this is quite important (spawning a bunch of cycles when you don't want to is problematic!).

[hjoliver]

After some thought, closing this as solved by cylc remove: for retriggering past sub-graphs, only the "removed" tasks will re-run.

In principle, we could want to prevent flow-on to subsequent cycles when triggering future flows too, but that's very niche (it's pretty hard to think of a valid use case) and I think it's OK to say that new flows behave just like the original flow - they do flow on to future cycles if they include parentless tasks or inter cycle dependencies.

@MetRonnie - feel free to reopen if you disagree.

[MetRonnie]

Ha, I see I got confused about this the year before as this was also a duplicate of #5555!