resolvers: run in separate processes

[oliver-sanders]

In multi-user setups we may have multiple users subscribing to multiple workflows simultaneously.

Large workflows can cause heavy server load in some cases (e.g. https://github.com/cylc/cylc-uiserver/issues/547). Because we handle each request synchronously on the same process, this means larger updates can hold up other updates. In the extreme, this can cause UIs to show as disconnected as a result of websocket heartbeat timeout.

Ideally we would find a way to run the resolvers for each workflow in separate processes to isolate them from each other. Though we would probably want to limit the number of processes and distribute subscriptions across a pool.

Original Post

We run subscriptions in a ThreadPoolExecutor.

In Python only one thread can actively run at a time (because of the GIL) so there is no compute parallelism advantage to this (but there may be IO concurrency advantages depending on the implementation details of the code being run).

This means that one large workflow can hog 100% of the CPU of the server, causing issues with other workflows.

We should be able to swap the ThreadPoolExecutor for a ProcessPoolExecutor. I had a quick try, but it didn't work first time, the first update came through but subsequent ones got stuck, so a little work required.

This does raise the question of how many processes the UI Server should be allowed to spawn. I think we should be able to run more subscribers than processes in the pool, but would need to read the docs to find out.