cylc-flow icon indicating copy to clipboard operation
cylc-flow copied to clipboard

Optional outputs can cause legit partially-satisfied prereqiuisites

Open hjoliver opened this issue 1 year ago • 3 comments

We need to analyze partially satisfied prerequisites before using them as an excuse to stall. If some correspond to optional outputs, then partial satisfaction may not indicate an error condition.

Simplified version of user workflow:

[scheduling]
   [[graph]]
      R1 = """
         FAM:fail-all? => bad
         FAM:succeed-any? => good
     """
[runtime]
   [[FAM]]
   [[a]]
      inherit = FAM
      script = true
   [[b]]
      inherit = FAM
      script = false  # OK, graph says success is optional for me 
   [[bad]]
   [[good]]

Result:

INFO - [1/good running job:01 flows:1] => succeeded
WARNING - Partially satisfied prerequisites:
      * 1/bad is waiting on ['1/a:failed']
CRITICAL - Workflow stalled

The problem is: bad depends on multiple optional outputs, so we should not be surprised that it ends up partially satisfied.

Currently the only automatic solution is ... our old friend the suicide trigger 😠

"FAM:succeed-any? & FAM:finish-all? => !bad"

(We have to wait for all members of FAM to finish, because bad could be spawned after good triggers).

hjoliver avatar Sep 14 '23 00:09 hjoliver