CLI: Upgrade auto-completion for Cylc 8

[oliver-sanders]

CLI auto-complete is broken.

UID makes it possible for us to complete much more including hierarchical workflow IDs, cycles, tasks and jobs.

Should be relatively straight-forward (famous last words).

See the cylc-admin proposal for details:

https://github.com/cylc/cylc-admin/blob/master/docs/proposal-universal-id.md

Quick notes:

• The previous auto-complete suffered from performance issues as it used the filesystem to list Cylc sub-commands. We can now load the commands via Python. We should investigate cashing these to improve performance.
• Where patterns are used (e.g. *) it would be good if the completion could auto-quote IDs to ensure the matching is being performed server-side.
• Suggest only implementing Bash for now 🐟 🐠 🎣.

Pull requests welcome!

[oliver-sanders]

Taking a quick look back at this one...

I managed to get a POC working which did most of what we needed it to. Unfortunately I can't remember where I put it :facepalm:.

The implementation worked well enough, however, ID parsing wasn't ideal and scan was slow (because it had to call a Python subprocess).

So why not write the completion in Python? Well, because we would need a Cylc CLI server, which would need authentication and whatnot. However, I had a crazy idea which remarkably works! Bash coprocesses. Coprocess allow you to pipe to and from a subprocess. Consider the pipe a | b | c, the coprocess would be b allowing the calling script to be both a & c.

Here's a quick example which uses a Python coprocess to compute square numbers:

#!/usr/bin/env bash    
       
coproc python cylc-completion.py    
       
for numb in $(seq 1 5); do    
    echo "# $numb"    
    echo $numb >&"${COPROC[1]}"    
    read var <&"${COPROC[0]}"    
    echo "\$ $var"    
    sleep 1    
done      

import sys                                                                           

                        
def listener(callback):    
    line = sys.stdin.readline()    
    while line:    
        print(callback(line))    
        sys.stdout.flush()    
        line = sys.stdin.readline()    
     
     
def responder(line):    
    numb = int(line)    
    return numb ** 2    
     
     
listener(responder) 

This would allow us to write the whole completion in Python which would make life a lot easier and allow zippier responses. It would also mean that we could support Bash, Fish, etc with a single completion script! Here's a Python completion script which does most of what we would want it to and took <20mins:

import asyncio
from pathlib import Path
from shlex import quote, split
import sys

from cylc.flow import iter_entry_points
from cylc.flow.id import tokenise, detokenise
from cylc.flow.network.scan import scan


COMMANDS = sorted([
    entry_point.name
    for entry_point in iter_entry_points('cylc.command')
])


def rdir(workflow):
    return Path('~', 'cylc-run', workflow).expanduser()


async def listener(callback):
    line = sys.stdin.readline()
    while line:
        print(' '.join(
            quote(item)
            for item in await callback(split(line))
        ))
        sys.stdout.flush()
        line = sys.stdin.readline()


async def responder(*items):
    length = len(items)
    if length == 0:
        return COMMANDS
    if length == 1:
        # command = items[0]
        return await list_workflows()
    if length > 1:
        tokens = tokenise(items[1])
        run_dir = rdir(tokens['workflow'])
        return await list_in_workflow(tokens, run_dir)
    return []


async def list_workflows():
    ids = []
    async for flow in scan():
        ids.append(flow['name'])
    return ids


async def list_in_workflow(tokens, run_dir):
    parts = []
    if tokens.get('cycle'):
        parts.append(tokens['cycle'])
    if tokens.get('task'):
        parts.append(tokens['task'])
    if tokens.get('job'):
        parts.append(tokens['job'])
    return [
        path.name
        for path in Path(run_dir, 'log', 'job', *parts).iterdir()
    ]


async def list_cycles(run_dir):
    return [
        path.name
        for path in Path(run_dir, 'log', 'job').iterdir()
    ]


async def list_tasks(run_dir, cycle):
    return [
        path.name
        for path in Path(run_dir, 'log', 'job', cycle).iterdir()
    ]


async def list_jobs(run_dir, cycle, task):
    return [
        path.name
        for path in Path(run_dir, 'log', 'job', cycle, task).iterdir()
    ]


# asyncio.run(listener(responder))

async def test(*input_):
    print(f'$ cylc {" ".join(quote(item) for item in input_)}')
    print(f'# {" ".join(quote(item) for item in await responder(*input_))}')
    print()


async def tests():
    await test()
    await test('trigger')
    await test('trigger', 'one//')
    await test('trigger', 'one//1')
    await test('trigger', 'one//1/foo')


asyncio.run(tests())

What's the catch? Well you would get one cylc completion command running for each shell that completion is loaded in. Is this a problem? The number of active terminal sessions should be small and the completions will be idle until requests are sent from the CLI? The Python processes would be fairly light.

[oliver-sanders]

[meeting 2022-06-27]

This is probably ok providing that we can limit the nnumber/lifetime of the auto-complete processes.

The suggestion was to start the auto-complete processes on-demand (i.e. when someone types cylc <tab>) and have them timeout after a set period of inactivity.

[hjoliver]

we could support Bash, Fish, etc with a single completion script!

Brilliant (Fish fan here).

Tried the example; that's very cool.