Request: a way to show tab completions

[wch]

First off, thanks for creating this package.

I'm using it create a REPL using Pyodide (Python compiled to wasm), and I would like to add support for tab completions. The idea is that when you press tab, it does one of the following (which is similar to the normal python REPL):

• If there are zero possible completions, then nothing happens.
• If there is one possible completion, it completes it.
• If there are two or more possible completions, then it fills in any following characters that are present in all of the possible completions, and then displays the possible completions.

I don't see a good way to do this with the existing API. One way to go would be to provide a way register a completion handler; this handler would be a function which takes a string (the text typed on the command line so far) and returns an array of strings (the list of possible completions).

So the usage would be something like this:

    readline.setCompletionHandler((text: string): string[] => {
      // Do stuff here
    })    

[strtok]

When I wrote xterm-readline I based it off rustyline (a readline library for rust), and rustyline does have completion support implemented here:

https://github.com/kkawakam/rustyline/blob/master/src/completion.rs

I can't remember why I never implemented it. I think there's probably different ways to implement completion and I'm not sure which way is best for xterm-readline.

I think what you've proposed seems sane. I wonder if the handler should choose which completion choose be the best candidate instead of returning the array?

[wch]

I like the behavior of the Python REPL, which appears to be the same as bash on my computer. (Maybe because both use readline?)

Suppose I start python and type ha:

$ python3
Python 3.9.13 (main, May 24 2022, 21:13:51) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ha

Then if I press tab one time, it prints out the possible completions and completes as many characters as possible, resulting in has :

>>> ha
hasattr(  hash(     
>>> has

So maybe the handler should return the current completion (like has) and an array of possible completions (like ["hasattr(", "hash("])?

[wch]

One more comment: now that I think of it, providing the current best completion and array of possible completions isn't a great option. It pushes complexity down to the user of xterm-readline, but it doesn't provide any benefit -- if the user types ha and the possible completions are ["hasattr(", "hash("], then the next letter must be s and there's no point in making the handler author write the logic to figure that out.

[strtok]

Yeah, I like showing the possible completion candidates to the user too but it's a bit trickier to implement because it requires rendering the list and then re-rendering the prompt on new lines and setting up the new cursor position.

I've noticed the python one will both auto complete and show the list depending on how much ambiguity there is.

For example, if you type "ha" and press tab python will auto complete automatically to "has" without showing candidates because the only two candidates are ["hasattr(", "hash("].

But then a second tab press (actually third? oddly the second one just sent a bell) shows the completion candidates.

[wch]

I've noticed the python one will both auto complete and show the list depending on how much ambiguity there is.

For example, if you type "ha" and press tab python will auto complete automatically to "has" without showing candidates because the only two candidates are ["hasattr(", "hash("].

But then a second tab press (actually third? oddly the second one just sent a bell) shows the completion candidates.

Interesting: that's not how it works for me. Pressing tab a single time both completes to has and shows the candidates. Same with bash on both my Mac and Ubuntu Linux machine. However, in a Docker container with Ubuntu, the behavior is the same as you describe: first tab does the partial completion, second tab shows the candidates.

It looks like the behavior is controlled by an option, show-all-if-ambiguous: https://www.gnu.org/software/bash/manual/html_node/Readline-Init-File-Syntax.html#index-show_002dall_002dif_002dambiguous

I personally don't have strong feelings either way, but I'd slightly prefer showing the candidates on the first tab press.

[strtok]

Interesting! Thanks for the investigation on show-all-if-ambiguous :)

I also prefer showing candidates on first tab press. It's intuitively what I expect to happen.

[strtok]

I was playing around with implementing this and it brought a lot of questions up. I think the primary question I have is what input the readline library provides the callback function. Does it provide the entire buffer?

GNU Readline seems to actually separate the "last word" of the buffer given separator characters, and then provides only the last word to the callback. I'm not sure I like this, partly because my use case for xterm-readline can tokenize and parse out the string to "complete" by itself.

Does it make sense that the callback should return both the parsed partial prefix being completed, and then list of candidates?

For example, in the 'ha' example above the response from the callback would be:

{
   "prefix": "ha"
   "completions": ["hasattr(",  "hash("]
}

This gives the readline l library enough information to actually auto-complete given any arbitrary grammar.

[wch]

I agree, I played around a little with this myself the other day and encountered the same issue. I am glad that you are looking at it, though!

From the perspective of the typical user of this code, it would be nice if xterm-readline did the following:

• Tokenizes the code with some reasonable defaults that work for most languages
• Invokes the callback with just the last token
• The callback would return an array of completions, or the data structure you suggested (I don't have strong opinion either way).

For most use cases, requiring the user to tokenize the entire current string would be a bit inconvenient, and would require extra thought about tokenizing, just to produce code that's essentially the same for most languages.

All that said, I get where you're coming from, and can imagine cases where it's useful to provide the entire string. In that case, the returned object you suggested makes sense to me.

So maybe: last token vs. whole string could be an option, and the callback would be expected to return the data structure with prefix and completions?

Or maybe, instead of last token vs. whole string, the user could provide a tokenizer function.

What does rustyline do?

---

One other thing I ran into: In my use case, I'm connecting this to Pyodide (Python in wasm) running in a web worker, so all of the communication is asynchronous. In my local copy, the setCompletionHandler callback is expected to return a promise, and where the completion handler is called from readKey, it looks roughly like this:

      case InputType.Tab:
        this.completionHandler(text).then((completions: string[]) => {
          this.term?.write("\r\n" + completions.join("  ") + "\r\n");
          this.state.refresh();
        });
        break;

But I'm not sure that it's safe to do this async stuff this way. At any rate, I hope that what you come up with works safely with an async completion callback.

BTW, if you're curious, this is what I'm using xterm-readline for: https://shinylive.io/py/examples/

[gajanak]

Hello,

it's some time since this Request, but I want to add my Question ;) IMHO:

I solved this by attachCustomKeyEventHandler which detects “tab” key up. In this, I take state.buffer() call a handleTabCompletionHandler(string):string[]. If this handler returns 0 lines — make nothing, if returns 1 line, state.update(line[0]) if more than 1 line, I print all this lines, call state.refresh() and restore the current prompt+buffer.

Now my question, is there any interest in a PR to have this simple tabCompletionHandler in this lib ? Would make my code simpler. And the perhaps complex logic of auto-completion can be done in the context of the use case, this lib stays clean.

If there is any chance to get this in Lib, I would try to create a PR for this ? (not much experience with ts and don't know much about problems with asynchronous stuff in this case) ;)

Update: Now, on reading this Request again and again. I noticed that this is the same as the idea of the first post. ;) It's possible to integrate this simple form first ? Perhaps callback could return (newBuffer:string|undefined,lines:[]string) to implement the good idea from comment number 4.

[youthug]

Hey guys, I'm trying to implement this in the forked repo: 6d99d865ce66cd0b18dfdf4b7bc6aef37dcee843.

Everything seems to be working fine (possibly), ~~except for one issue: mine: mongosh: The printed candidate input suggestions appear a bit messy. I'm not sure how to handle it. https://github.com/youthug/xterm-readline/blob/6d99d865ce66cd0b18dfdf4b7bc6aef37dcee843/src/readline.ts?plain=1#L302-L320 Any suggestions?~~

Update: It has now been better printed (following the example of mongosh):

However, I have not done a comprehensive code testing yet, and the code implementation may not be optimal. Any suggestions are welcome!