CHECK_CIRCULAR_LIMIT

[oliver-sanders]

The cyclic dependency check and by extension the un-satisfyable dependency check (which is invoked via get_graph_raw) are currently only performed when the number of tasks in the workflow is less than 100.

https://github.com/cylc/cylc-flow/blob/e4659ad20d713858d3c9802715e9b20bdc80a27e/cylc/flow/config.py#L164

The reason for this is that we had previously encountered workflows which took a long time to complete these checks, the number 100 is completely arbitrary. Note in a validate-install-play user pattern the circular dependency check is currently run twice (during validate & play). These slow validation results were spotted with older versions of Cylc 7 and have not been reproduced with Cylc 8 / Python 3.

We should test cylc validate against the workflows we can get hold of to find out how much resource the most demanding workflows are currently taking. Toggling --check-circular on and off should give us an idea how much burden it is. We can most likely bump the number up from 100 or make this test looser in some way. Note I would expect get_graph_raw to scale with the dependency density within the graph window evaluated, the number of tasks is just an easier metric.

Pull requests welcome! This is an Open Source project - please consider contributing a bug fix yourself (please read CONTRIBUTING.md before starting any work though).

[hjoliver]

Note I would expect get_graph_raw to scale with the dependency density within the graph window evaluated, the number of tasks is just an easier metric.

As I recall, get_graph_raw returns the whole graph between two cycle points, Cylc 7 stylz.

For the Cylc 8 graph view we'll need to generate n-window graphs around active tasks, regardless of cycle point.

In current validation we use get_graph_raw to generate the first 3 cycle points of the graph, and check that for circular dependencies. So here the error gets detected if final cycle point = 3 but not if final cycle point = 4:

[scheduler]
   allow implicit tasks = True
[scheduling]
   cycling mode = integer
   initial cycle point = 1
   final cycle point = 3
   [[graph]]
      P1 = "a => b => c => d => e"
      R1/$ = "e => d"  # circular at $

What we should do, for the circular check, is generate the whole graph for the first point of every recurrence?? Hmm, no, even in my example that wouldn't work because its the combo of the P1 and R1 graphs that causes the problem.