reduce the impact of FS scans

[oliver-sanders]

At present we are regularly performing file system (FS) scans to detect new suites. This functionality is required to pick up on suites started from the CLI.

These FS scans are quite regular and can be slow as they involve walking the directory tree.

• Is there a more light-weight alternative to a full FS scan?
  • To pick up new suites we only need to listen for directory creation.
  • However, due to hierarchical suites these directories could be created in different places.
  • Note a solution cannot be Linux-bound (though adding something like fswatch to the dependency base shouldn't be an issue)
• Can we do some kind of incremental scan to avoid re-walking trodden ground?
  • Perhaps an arrangement of "quick scans" and "full scans".

Note: For suites started via the UI we can do things in a more responsive manner (https://github.com/cylc/cylc-flow/issues/3451).