Proposal: Freeze the completion API

[maxlandon]

Disclaimers

First and foremost, I must say that this proposal, any sound solution to it and all the potential questions and problems raised on the way of finding one, are probably all well above my paygrade. The enormous reliance and trust of so many projects on cobra can only add weight to my statement.

None of the repositories involved in this proposal are mine nor ones I have made major contributions to, so I will strive to formulate what I perceive as necessary changes to operate, questions to answer or debates to open, with the outmost respect and consideration for the authors and contributors of all of these libraries, for I'm already very grateful to use their code.

Therefore my principal aim and hope about this (as far as I can see it) is at least to gather some attention around the topic being exposed below. As importantly, I hope that this may at least give hints or ideas to people, or even answers to some technical aspects, even if these answers are not favoring any of what follows.

Finally, I will say that I'm still proposing this because in my own and very, very humble view, the benefits of this proposal ultimately will outweight its costs. But its my guts and my brain.

Introduction

It is pretty much safe to say that cobra is the most widely used and mature CLI library. As well, that it has been the first library to figure out - and succeedingly implement - how best to deliver completion code out to users' shell environments: let the tool describe itself, with no need for more than a couple of shell function builtins to interface with users (in either direction: cobra completion shell snippets don't need many builtins from any shell, and users don't have to do much more than sourcing a script).

The use of __complete has thus proven immensely useful to many, with very few downsides: The obvious first one is that it has to stick with the initial model and that enhancement in shell completion functionality and logic would have to be on the shell-side (in other words, that opening the cobra API too much for the mere sake of completions is very often not a good or acceptable tradeoff). This last point is one of these cores around which this proposal revolves.

The _git completion file in standard Z-Shell tops 8K LOC. I won't risk myself estimating how much less it would be had git been written in Go today, and still the benefits would be multiplied by 4 (supported shells for completion). This reminder is not a comparison. On the contrary, it's an acknowledgement of the fact that things have greatly evolved in the last 10 years in the CLI space, that some lessons from the past have been learned and new ideas/approaches have succeeded.

More recent iterations

When having to write a console application library for some tool -and after having passed, in this respect or in others, a significant amount of times burried in completion shell scripts-, I stumbled upon the carapace completion engine, which some of you may already know.

For those who don't, carapace takes the idea of the __complete command several steps further. Benefits include - but are not limited to - a support for more shells, more efficient shell scripts and sourcing, better functionality per line of code (shell and Go), and obviously the luxury of focusing on one thing (completion), so being free of expanding the related API (all other things kept equal, obviously).

Having used this engine in several of my projects, across several shell/Go runtimes, I must and will happily attest of the outmost reliablity of the library, its incredible ease of use and combinatory power. This engine and its API drastically reduce the feature-set/reliablity to dev-time ratio for even the most complex completion tasks, turning something that would take (1-hour x 9 shells) shell script development session into a 5-min job done once and running on all of them. The laudatum could continue on several other aspects and features. The amount of work put in it is big with regards to its manpower (1 person).

Not for the sake of advertisement, I must emphasize that I could use both cobra and carapace in this console closed-loop shell flawlessly, with big a command tree and large functionality, by patching only ten lines of Go code, which can be removed if one specific issue below is solved. Two libraries not made for closed loop + a readline shell and a patch = all fun no bugs.

Old programs to new ones: what changed

Consider the following two things:

• There is an entire generation of the-world-running-on-tools with C codebases (git and ffmpeg are some examples that I wil use in this section). It is paradoxical that they often have a very large and "intimidating" CLI, with various options and arguments whose arguments can be much more evolved than what the --help flag suggests (despite all good intents).
• On the other hand and coming decades after this first generation, consider widely used modern CLIs like kubectl, helm, etc. We can witness how cobra being an exhaustive CLI library has sometimes prompted developers to vastly expand their tools' CLI, because suddenty, building more structured and recursive command trees was much easier.

Here again, there is a risk of having an excessively large CLI, doubled by the sometimes not obvious need to maintain strict backward compatibility for it (especially when invocations are found within automated, configuration-as-code style scenarios).

My point here is again not to make a side-by-side comparison of tools coming from different eras. Had cobra existed as a C library, maybe most of the aforementioned tools -or equivalents- would not have been written the same way. However, I suspect that there is an opportunity not to find oneself in this paradoxical "discrepancy" between some large programs' toolsets and their capacity to offer this toolset at various stages of the CLI lifetime with equal power.

Proposal & Suggestions

However scary that might appear, I think my proposal is no more than what this issue title says: freezing everything in the API that is related to completions, and "pass it into maintenance mode".

That would grossly include:

• No more ShellCompDirective constants added.
• No more shells support added.
• No functions related to flag/arg command completions added.

This does not mean, however:

• Removing support for any of the currently supported shells.
• Removing any of the directives.
• Stopping support for the existing completion shell script code.

Important questions & problems

• Whether or not to integrate carapace code into the cobra module, under a separate directory. This is the kind of question that is way, way above my paygrade and stake in this. And I think doing this would have almost no benefit at all, in addition to many people legitimately arguing that they don't want such a dependency. But I might be wrong, and I suspect anyway that some people might ask this to themselves too.
• What mechanism to use so that all calls to the current __complete command can be safely "bridged" to carapace if its present/needed, or if the command should be overwritten altogether. This area of things will probably expand into a larger set of questions/approaches that I currently don't have at mind.
• There are some other questions that I either don't have at mind now, or that I've never had (which is a given).

Targeted End State

• Any new comp code going forward for any given software might freely choose between either sticking to the legacy cobra completion API because it is enough for their use case/constraints, using the carapace library for any new features, or any mix of both with equal support.
• Any user having carapace installed will beneficiate from it whenever he is using a cobra program, transparently and without having to know it. Besides, anyone not having carapace should see no change from before.

Advantages

• Redirect as many persons interested in completion/hint/whatever support to solve issues on the carapace repo, dedicated to this very problem. This is possible not only because the latter is quite bug/issue free, but also because using carapace gives an additional layer at which problems can be solved. And if really the issue could not be solved at this library level, then one could bother the cobra devs for something/hint/fix. See the PR/Issues sections below for examples.
• Win-win for everyone: cobra devs can focus on the core library, carapace can take care of at least 80% of completions feature/maintenance. Users get both of it at will and at the dose of their choosing.
• Obviously, carapace currently has support for 9 shells, and if I'm not mistaken, each and every shell supported in cobra does not come close to the quality of carapace. Opiniating again...
• carapace having for core principle of being identical in behavior for all shells to the extent that this is possible, it also tends to raise the lowest common denominator for all aspects of the program completion/display on all shells. In turn, shells may want to improve themselves because they happen to be weighting down on this denominator.
• carapace already has support for exported completions and a restrained but smart set of metadata destined to shells for them to consume all this completion output.
• Support for flag types, arguments, positional arguments of all sorts and origins is fantastic.
• carapace enable usage of the OS environment variables, gives access to the entire completion context at any point in your code, and much more related to this.

Disadvantages

• carapace is currently a distinct binary, that needs to be installed separately from cobra. It currently also requires a sourcing call in users shell for activation, and also naturally one for each program being installed that wants to complete itself. 1+(n programs) is still one call more than what is currently required. Related to the Important problems section. I might balance that disadvantage by opiniating that completion sourcing is much more intuitive on the aggregate with carapace, but that's an opinion and nothing else better.

Requirements / Steps

• CompletionOptions might need to be considered/ported/refactored/integrated into carapace.
• Verify compatibility with a representative set of programs: enroll some programs on volunteering ?
• Debug functions/code : maybe not all of it needs to be ported, or could be in more idiomatic way.
• Ensure that the carapace execution workflow is entirely compatible with the current state of things, and ensure its API for run/pre-run is sufficiently consistent for long-term compatibility.

Related issues / PRs

Below is a list of the open issues and PRs I have found to cross the proposal at hand at some point. Some of them raise important concerns that have sometimes not been answered/settled on here, but where carapace has.

PR/Issues on cobra repo

• #1529: This issue touches on the problem of sourcing shellcomp code into shells, the cost of doing at various points and with various methods, as well as the effects of program binary updates on shells' ability to properly ingest its contents.
• #1629: Note to @rsteube that I've never had this use case so I don't know if this works by default in carapace.
• #2012: This is quite an important one with many regards. First, I'm damn sure there is a way to avoid global state for this map, and also pretty sure that the last attempt to do so has made a very little mistake that no one seen. What is important here, is that anyone reading this very line is very unlikely to realize how much potential is held hostage by this very little problem. So. much. potential.
• #1943: This is directly related to #2012, and should be solved conformingly with the latter's fix.
• #1732: This would be a solved problem.
• #1743: This one also.
• #1833: See carapace handling of this with its PrefixMatcher (again, compatible with all shells)
• #1856: Already solved by carapace on all shells.
• #1857: Completion support for nushell could be redirected at the carapace repo: there is already support for nushell, and it might quicker to port what is missing as compared with this PR.
• #1867: Maybe this can be solved within the work of this proposal.
• #1922: ActiveHelp is already supported in carapace, and its API for hints/helps is cleaner.
• #1977: Already solved in carapace.

PR/Issues on other repos

• #655: Please see the first answer given by @rsteube to my PR, which pretty much sums up what is holding carapace from implementing full "backend/fallback support" for all completions declared through the cobra API, which will allow anyone importing carapace in place of / next to the completion command to beneficiate from past completion code out of the box.

Conclusion

Go developers are already very lucky to have these existing projects at their disposal, which may not be said of many other languages and their stacks. All of these projects are going in wonderful directions and have pretty much all taken smart and rigorous approaches to their own problems, providing huge impact.

I'm pretty sure that a very little bit of synergy and separation of concerns will enable more things than most would suspect. In the hope that this will raise some thoughts and initiatives !