Dynamic completion support

[Manishearth]

Maintainers notes:

• Assuming this is custom completion on top of clap-driven completion, this is blocked on #3166
• Open question: does the completion function accept just the arg and current value or does it have access to ArgMatches for completing based on other data like in #1910

Like with PossibleValue::help, we should support help text

---

Currently clap does basic completion of arguments, however it can't be used to do things like the dynamic completion of git checkout <branch>.

It would be really neat if this could be handled by clap itself -- AFAICT nothing like this currently exists for any language and it would just be magical since you can then do everything in Rust code.

The rough idea is that the Arg/Subcommand builder can take a .with_completion(|args, str| ... ) closure. We pass the closure the partial parsed list of other args, and the partial string being completed for this arg (may be empty). The closure returns a list of completions.

When you attempt to do something like git branch -D foo<tab>, the completion script will call git --secret-completion-param 10 -- branch -D foo, where 10 is the index in the arguments of where <tab> was pressed, and after the -- we pass all the arguments. Clap parses these arguments except the partial one, and passes this down to the with_completion closure, which will return a list of completions which we print out (and feed back to bash or whatever).

A simpler version doesn't handle parsing all the other arguments, instead your closure just takes the partial string, so you just provide a closure operating on a string.

If this could be made to work neatly, it would be pretty great.

cc @killercup

[kbknapp]

Sorry for the late reply, it's been a busy week!

This is related to #568 and something I very much want. The implementation you've outlined is pretty close to what we're thinking in #568 and probably almost exactly what will end up being implemented. I haven't worked out the final details yet as I've had other issues as as priority and the v3 release is a prerequisite for me to implement this (although I want this, or an initial implementation in the v3-alpha1 release).

[sportfloh]

hey, I was recently looking for such a feature and stumbled over clangs autocompletion feature (http://blog.llvm.org/2017/09/clang-bash-better-auto-completion-is.html). Maybe their solution is somehow inspiring. Can I help with this issue?

[kinnison]

To reiterate a call for this, we'd love to have this in rustup so we could offer more localised completions (e.g. for +foo to complete over installed toolchains).

[woodruffw]

I'd also like to register interest in this. It's not as conceptually clean, but an approach that injects the results of a subshell would be sufficient for many CLIs.

Borrowing the with_completion terminology above:

subcommand(
  App::new("dump")
  .arg(
    Arg::with_name("thing")
        .with_completion("foobar list")
  )
)

would go from something like this in the bash completions:

opts=" -h -V  --help --version  <thing> "

to this:

opts=" -h -V  --help --version  $(foobar list) "

(N.B. that this approach applies to Args and not the subcommands themselves).

[CreepySkeleton]

I must admit I like the "special option to autocomplete" approach very much, but I've also came up with a number of tricky questions.

Let me summarize the prior discussion and outline the desired solution that should work and satisfy everybody.

We want this facility somewhere in clap:

fn dynamic_completion<F>(hook: F) -> Variants 
where 
    F: FnOnce(Input) -> Variants;

Where to squeeze the fn in? I believe it should be a method in App, and this method should only affect the current subcommand by default. Developers are supposed to call this method for every subcommand separately. We may also provide a "global" variant.

The hook here is effectively a piece of developer-written code whose job is to look at the args that have already been supplied and provide the completion variants back to clap.

How it works: presence of this kind of hook tells clap to generate a special --autocomplete option. The binary simply prints the completion variants, one per line (see example below).

How does clap supply the variants back to the shell so the later can use it? It doesn't. What it does is printing the values to stdin. This option is supposed to be called from inside the completions script, like that. The script will handle the rest in a shell-specific way.

What does Variants look like and why not using simple Vec<OsString>? Because we need to leave the ability for devs to fall back to the default completion script, see also https://github.com/clap-rs/clap/pull/1793 to get what I mean by "default completion".

// we may want to extend the system in future
#[non_exhaustive]
enum Variants {
    /// Return this when the completion hook couldn't decide on the completion.
    /// This will fall back to the default completion system which should work sensibly.
    Fallback,
    
    /// Return when the hook did found some good options to complete.
    Success(Vec<OsString>) 
}

Some examples:

# What user does
cargo +<TAB>

# the call 
cargo --autocompletion 1 '+'

# the output (OK, pack the values and return early)
SUCSESS
nightly
stable
beta

# ATTENTION! There's a space between `+` and `<TAB> 
cargo + <TAB>

# the call
cargo --autocompletion 2 '+' 

# the output (Oops, let's just cross fingers and let the script handle the rest)
FALLBACK

The next question is: what is Input. Well, we need to

1. Allow devs to specify the name of the long option
2. Trim everything prior to the actual args for the current subcommand, if any
3. Tell user if there was a space between the last arg and TAB

fn dynamic_completion<F>(name: &str, hook: F) -> Variants 
where 
    F: FnOnce(bool, &[OsString]) -> Variants;

.dynamic_completion(|has_space, args| { /* ... */ })

// alternatively, see |args, str| design in the first comment

Does everybody agree with this design? Any unresolved questions?

[Manishearth]

Yeah, this is essentially what I proposed above, with the small tweak that I was using -- as an argument separator instead of quoting the entire thing (I find quoting to just be very messy), and also I was havng the completion function live on the Arg. I do think having a completion function on the Arg (and one that lives on the whole command) is good: this way it is very easy to write dynamic completion for just one argument, but you can also write it for the whole function if you want.

I'm less fond of the has_space, args approach because the user needs to reparse the args in this case. IMO we should parse the rest of the args and surface them to the user in a structured way, and then surface the partial string being tab-completed.

Trim everything prior to the actual args for the current subcommand, if any

I think it should be all args except the one currently being tabbed. It should be okay to tab-complete in the middle of an arg list

[CreepySkeleton]

instead of quoting the entire thing

No quoting the entire string here (we simply don't know what the string was with most shells). We pass the args array as is, quoting each arg to guard against spaces. But using -- is probably a good idea so, if user used -- on his own, it would be accounted for.

# what we do
app arg1 --flag --opt1 val1 'val 2' -- something <TAB>

# what we get
app --autocomplete 7 -- '--flag' '--opt' 'val1' 'val 2' '--' 'something'

user needs to reparse the args in this case

User needs to reparse all args in any case. The "partial parsing" problem was reported as https://github.com/clap-rs/clap/issues/1880, and I think this could work as a viable option. But anyway, the hook should work on top of bare strings and expect partially_parsed_matches to be incomplete and/or not quite correct.

So, maybe:

// or maybe index instead of has_space?
.dynamic_completion(|has_space, partially_parsed_matches, args_as_os_strings| { /* ... */ })

[kenoss]

A few weeks ago I saw completion approaches of git, kubectl, pyenv and clap_generate and I found that dynamic completion will be suitable for clap.

I agree with th CreepySkeleton's first observation. But, I vote Manishearth's point. I prefer the completion function live on the Arg. I think forcing users to parse args themselves decrease clarity of clap. We should provide structural way if we can.

I expect normalization works for #1880 .

Is there a case one have to parse raw string? Let me classify the cases.

Simple cases.

# Candidates are options of `branch` subcommand.  `App("branch")` is responsible for completion.
$ git branch -<TAB>

# Candidates are short options of `branch` subcommand.  `App("branch")` is responsible.
$ git branch -a<TAB>

# Candidates are branches.  `Arg` with index of `App("branch")` is responsible.
$ git branch -a <TAB>

# No candidates.  `Arg` with short name `m` is responsible.
$ git commit -m <TAB>

More complex cases. Note that completion of git 4 uses variables:

• cur: current word, word arround <TAB>
• prev: previous word
• words: array of arguments
• cword: num of arguments

# `--strategy` and `--strategy-option`
$ git cherry-pick --strategy<TAB>

# Candidates are strategies.  `--strategy` is responsible.
$ git cherry-pick --strategy <TAB>
$ git cherry-pick --strategy=<TAB>
$ git cherry-pick --strategy <TAB> foo
$ git cherry-pick --strategy=<TAB> foo

The code is here. This uses cur and prev.

Clap version will be done easily because parser will remove ambiguity of and = and we get candidates by asking Arg("strategy").

$ git fetch <TAB>
benches/        clap_derive/    etc/            ip6-localhost   localhost       src/            tests/          upstream
clap-perf/      clap_generate/  examples/       ip6-loopback    origin          target/         trochilidae

$ git fetch --mirror <TAB>
cow          destringify  master-orig  staging      trying       v2-master
debug        master       release      staging.tmp  v1-master    v3-dev

$ git fetch origin:
(nothing)

$ git fetch --mirror origin:
HEAD    debug   master

The code is here. This uses subcommand's name and investigates its options. Note that git has subcommands at most depth 1.

Consider the last case. Clap version will be done by querying subcommand's name and options from Arg.

---

Seeing git's cases, I feel parsing by clap + Arg.with_completion + some query mechanism is sufficiently powerful.

[kenoss]

Note that clap can automatically derive completion for short/long options in many cases, i.e., if one doesn't need special treatment.

[PoignardAzur]

Is there progress on this issue? The approach suggested by @CreepySkeleton seems pretty good to me.

If the issue is stalled for need of a contributor, I'd be interested in spending a few days on it (especially if I can get some mentoring). I'd really like to improve cargo's autocompletion.

[pksunkara]

This is still in design phase. According to the current suggested design, this needs more foundational work regarding partial parsing (#1880).

[kolloch]

@CreepySkeleton I've been playing around with the code and I think we could get it to work to support dynamic completion of a simple flag value. I haven't looked at the shell completion logic but I am thinking in the direction of only calling your_app --autocomplete 7 ... if we are at a flag position with automatic expansion.

It would be super cool, to reduce all shell autocomplete logic to calling out to your prog with the --autocomplete argument but that would probably require a different parser or a lot more work.

Anyways, if we use the --autocomplete "API", we can step by step expand it to cover more use cases.

[blaggacao]

how cobra does it:

• https://github.com/spf13/cobra/blob/master/bash_completionsV2.go
• https://github.com/spf13/cobra/blob/master/zsh_completions.go
• https://github.com/spf13/cobra/blob/master/shell_completions.go
• https://github.com/spf13/cobra/blob/master/powershell_completions.go
• https://github.com/spf13/cobra/blob/master/fish_completions.go

and here is the 'hidden' command:

https://github.com/spf13/cobra/blob/56060d19f88533a20f635116f64aff2065cf260a/completions.go#L148

looks like this is becoming a 'copy paste' task

[nrdxp]

I would really appreciate this feature. Right now my app has a Struct describing what are basically inputs to all of my cli subcommands. It would be super great to use this information in completions. I was a bit surprised when I found out it may not be possible :disappointed:

[sergiimk]

As a long-time watcher of this issue thought I'd share my solution.

The CLI tool I'm working on deals with a lot of different kinds of entities that have long names (think kubectl), so dynamic completions were essential for usability.

The solution pieces are:

• Bash / Zsh completions delegate all work to myapp complete -- <words> <position> sub-command
• The complete command combines the input and a regular clap::App definition to establish the context (which sub-command we're on, are we completing a positional or a named argument etc.)
• Using completions is as simple as this

It's not perfect (e.g. has troubles with multiple positional arguments of different kinds, and I struggled to make it work with fish) but satisfies 90% of what I need and can be extended.

For Rust apps that are very fast to start I really can't think of a good reason to generate completion scripts instead of delegating the entire work to the app itself in all cases. Dealing with completion differences between various shells is no fun.

[chisui]

Haskells optparse-applicative has builtin completion support. Maybe a look at it's implementation is helpful.

[woodruffw]

This isn't a catch-all solution, but: perhaps clap::ValueHint could be extended to include a variant that enables dynamism through a subprocess?

What I'm thinking of is something like ValueHint::Process(String), where the variant value would be something like "foo --bar" and the shell completion generator would be responsible for emitting something like $(foo --bar) and correctly splitting and wrangling the resulting output into a completion set.

This would be a reasonable solution for many CLIs, and requires relatively little complexity on clap's side (it's just another variant, and generators would be responsible for emitting it or a fallback appropriately.

Thoughts?

[epage]

#3022 was a tipping point for me in realizing that maybe our current approach to completions doesn't work. We effectively have to implement a mostly-untested parser within each shell. Examples of other problems that seem to stem from this:

• https://github.com/clap-rs/clap/issues/2729
• https://github.com/clap-rs/clap/issues/2750
• https://github.com/clap-rs/clap/issues/2145
• https://github.com/clap-rs/clap/issues/1822
• https://github.com/clap-rs/clap/issues/1764

If we take the approach of argcomplete where we do the parsing in our core code, rather than in each completion script, this will help us share parsing logic between shell, share some or all parsing logic with clap itself, and make a subset of the logic more testable.

In my mind, this is raising the priority of this issue.

This doesn't mean we'll work on it immediately though. We need to first get clap3 out and some work that is spilling out of clap3. We also need to decide whether to morph the existing parser into supporting this or create a custom parser (since the needs are pretty special). If we do a custom parser, we should probably do https://github.com/clap-rs/clap/issues/2915 first so we can reuse lexing logic between clap and the completion generation. I've also been considering https://github.com/clap-rs/clap/issues/2912 which would allow reusing the completion logic with any CLI parser.

[AndreasBackx]

Hey @epage, I noticed your brainstorm #3476 and relevant issues about the modularisation of Clap, and also your comment on rust-lang/cargo#10472 about perhaps making this your next Clap priority. I would love to help out with maybe an RFC or implementation, though maybe it's better to ask where a newcomer could be helping out with this issue? Let me know whether you already have any ideas and I (and others) could perhaps share some ideas as well, like already done above.

Additionally, are there any other communication channels that are more direct? The old Gitter chat does not seem to be active, there doesn't happen to be some chat channel elsewhere?

[epage]

The first step is pulling out the lexer for reuse in the completions. I'm working on that now. Once thats in a good enough state, I can pull it out into a crate and we can create a proof of concept for this. At that point, its filling out features which will be easy for anyone to contribute to.

[epage]

@AndreasBackx I've got the start for #3166 in #3656. Once I have it merged, want to take over flushing it out so it sufficient for existing shell completions (cargo's handwritten or existing clap_complete users)? Once we are at that point, we can look at resolving this issue.

Additionally, are there any other communication channels that are more direct? The old Gitter chat does not seem to be active, there doesn't happen to be some chat channel elsewhere?

Sorry, forgot to respond to this one. We do have a zulip instance that we've not really advertised past the maintainers yet. I can also be reached on the rust-lang zulip strean "wg-cli". I'm also on both rust related discord instances but I tend to miss notifications there.

EDIT: I've also gone ahead and joined "kbknapp/clap-rs" on gitter