efficiency: investigate bottleneck

[oliver-sanders]

See also: https://github.com/cylc/cylc-ui/issues/1614

This workflow has proven to be remarkably difficult for the UIS & UI to handle:

#!Jinja2

{% set members = 10 %}
{% set hours = 100 %}

[scheduler]
    allow implicit tasks = True

[task parameters]
    member = 0..{{members}}
    fcsthr = 0..{{hours}}
  [[templates]]
    member = member%(member)03d
    fcsthr = _fcsthr%(fcsthr)03d

[scheduling]
  initial cycle point = 2000
  runahead limit = P3
  [[xtriggers]]
    start = wall_clock(offset=PT7H15M)
  [[graph]]
    T00,T06,T12,T18 = """
        @start & prune[-PT6H]:finish => prune & purge
        @start => sniffer:ready_<member,fcsthr> => <member,fcsthr>_process? => finish
        <member,fcsthr>_process:fail? => fault
      """

[runtime]
    [[sniffer]]
        [[[outputs]]]
{% for member in range(0, members + 1) %}
    {% for hour in range(0, hours + 1) %}
            ready_member{{ member | pad(3, 0) }}_fcsthr{{ hour | pad(3, 0) }} = {{ member }}{{ hour }}
    {% endfor %}
{% endfor %}

For more information see: https://cylc.discourse.group/t/slow-load-of-cylc-workflows-disconnects/823/19

Investigation so far has confirmed:

• The scheduler is not a source of delay.
• The UIS chokes on the update for several seconds.
  • During this time, updates to other workflows are suspended
• The UI chokes on the deltas of several seconds.
• The browser takes a couple of seconds to update.

This issue focuses on the UIS side of things.

Suggested remediation (Flow/UIS only, please update with new suggestions):

• https://github.com/cylc/cylc-uiserver/pull/548
  • Optimize a routine in the graphql-ws library which was causing ~11s of CPU time.
• https://github.com/cylc/cylc-uiserver/issues/538
  • Allow updates for other workflows to get through whilst the UIS is choking on a big update.
• https://github.com/cylc/cylc-uiserver/issues/545
  • Switch to a faster async framework.
• https://github.com/cylc/cylc-uiserver/issues/333
  • Update our graphql dependencies (currently blocked).
  • This would allow us to move to more recent Python versions and take advantage of improvements in graphene/graphql-core.
  • This requires dealing with some trouble dependencies which is getting quite urgent anyway.
• https://github.com/cylc/cylc-flow/issues/5907
  • A small cylc-flow side change to make the resolvers cylc-uiserver imports slightly more efficient.