RFC: Change the Provider API to allow LSP compliance

[illright]

Summary

Revise the Provider API in a backwards-compatible way to support the autocompletion features provided by the Language Server Protocol.

Motivation

The current Provider API has a number of inconveniences as well as limited flexibility. Most of the optional autocomplete features offered by language servers cannot be properly supported with the current API, as well as some mandatory features, which means that it's impossible to use language servers to provide proper autocompletion, one of the shiniest places of LSP.

The missing features are (most wanted first):

1. allowing the acceptance of a completion item to change text after the cursor as well (this is needed to support the two common modes of working with autocomplete these days: insert and replace. The former one turns cons|ole into const|ole and the latter one turns it into const|)
2. presence of a generic data field to allow for storing identifiers for completion items which could later be used for requests to resolve details on a particular item
3. custom text to use for sorting and filtering items in the completion list
4. commit characters (for example, in this.|, with a method completion doThings a commit character could be (, typing it would accept the selected suggestion and also insert the commit character, yielding this.doThings(|)
5. officially supported Markdown descriptions of completion items (I think this feature is currently experimental and thus not documented)

The aim is to create a next-gen API with as much backwards-compatibility as possible that would allow sufficient flexibility to implement truly smart autocompletion. That would be a major step forward to keeping on par with VS Code in providing a comprehensive development experience.

Describe alternatives you've considered

• Forking autocomplete-plus and implementing this API in a community autocomplete package.

This is always a possibility, but it is undesirable due to the need of distributing the alternative package to users and lack of performance benefits from having a package included in the V8 snapshot.

• Adapting the LSP protocol to the current API.

This has been attempted with varying success, but most of the benefits from having a dedicated language server aren't achievable. The lack of flexibility in the API is also a cause of some of the current issues to this repository, introducing a more flexible API could help resolve them.

Additional context: Proposed API changes

Provider object

{
  // all the same fields as currently
  filterSuggestions?: boolean = true,  // already present
  sortSuggestions?: boolean = true,  // new field
}

Suggestion object

interface TextEdit {
  newText: string,
  range: Range,
}

interface SuggestionBase {
  displayText?: string,
  sortText?: string,  // new field
  filterText?: string,  // new field
  replacementPrefix?: string,  // this field is ignored if `text` or `snippet` are passed the `TextEdit` object
  type?: string,
  leftLabel?: string,
  leftLabelHTML?: string,
  rightLabel?: string,
  rightLabelHTML?: string,
  className?: string,
  iconHTML?: string,
  description?: string,
  descriptionHTML?: string,  // new field
  // `descriptionMarkdown` field is deprecated
  descriptionMoreURL?: string,
  characterMatchIndices?: number[],
  additionalTextEdits?: TextEdit[],  // new field
  commitCharacters?: string[],  // new field
  preselect: boolean = false,  // new field
  data?: any,  // new field
}

interface TextSuggestion extends SuggestionBase {
  // this field accepts TextEdit objects now, backwards-compatible change
  text: string | TextEdit,
}

interface SnippetSuggestion extends SuggestionBase {
  // this field accepts TextEdit objects now, backwards-compatible change
  snippet: string | TextEdit,
}

New behaviour description

The main change is allowing the text and snippet fields to accept TextEdit objects, which contain not only the text to insert, but the exact range in the document to insert it in. This range can go behind and in front of the cursor, which allows free text modification.

By default, Autocomplete+ takes care of sorting, filtering and highlighting matched indices among the completion items. The latter can be overriden by the characterMatchIndices indices option (but it's important to have the default behaviour there and documented. May this is the case right now but due to the lack of documentation on this matter I can't tell). If the provider wishes to take over sorting and filtering as well, they may signal this intention with the sortSuggestions and filterSuggestions provider options respectively.

UPD: Suggestions have a preselect property which gives them precedence in sorting. Only used if the sortSuggestions option of the provider is set to true

Upon selecting a completion item in the list, the getSuggestionDetailsOnSelect callback is called with the suggestion object carrying the data property. When the completions dropdown is shown, the newly typed characters are checked against the commitCharacters array of the currently selected item. If any of those characters were typed, the suggestion should be automatically accepted and the commit character should be inserted right after the newly added text.
Note: the commit characters behaviour is off by default, but can be turned on in settings. That way we retain the current behaviour and do not break anything.

Upon confirming an autocomplete suggestion, in addition to executing the main text edit, any additional text edits are also executed. This should be atomic to allow to undo the whole change brought by the completion with a single Undo command.
Note: the default additional edits behaviour is to not apply them, but one can turn on applying them in settings. That way we retain the current behaviour and do not break anything.

Autocomplete+ doesn't perform any Markdown parsing/rendering, instead allowing providers to supply HTML strings. That way the providers could use their own Markdown parsers which are supported by the language server providing Markdown strings.

[illright]

The Atom community (me or someone else maybe) is willing to make a PR to implement these changes if there's a green light from the core Atom team.

[sadick254]

Hey @illright Nice RFC writeup :+1: . I agree that this would be a huge improvement to autocomplete in Atom. Having a backward compatible change that would allow LSP autocomplete features is definitely the way to go. Let me know if you need any help on this. I am looking forward to the implementation.