Completion by words in all loaded documents

[estin]

Hi! Draft: completion by words in all loaded documents

Works for document

• with language server support (merge lsp result and finded words)
• and without it

May be such feature must be implemented via plugins in future?

[the-mikedavis]

See https://github.com/helix-editor/helix/pull/2403

[estin]

Current solution gather completion options from all loaded documents/buffers and merge it with language server response.

Very often langauge server don't give me suggestions on sophisticated python code (too many magic and meta programming). And don't works on html templates and so on.

I'm create this PR as draft, because know it not so optimal yet. But it works well for my cases.

For my happiness I need on helix only ths things:

• recent used buffer picker #2980
• completion by loaded buffers, this PR
• on lost focus hide cursor or dim all view

Sorry for my poor English and not usefull noise here

[archseer]

#2403 has a better implementation but I'm still concerned about performance.

[estin]

New approach. Draft not fully implemented yet. Introduced worker.rs - separate thread to extract and filter words for completion (nothing special or new).

workflow

• on editor initialization starts Worker
• on first document load, it's content sended to Worker
• on text typing collects prefix and send it to Worker
• for language server merge items for completion

todo:

• on changes send it content to Worker to extract words (only new inserted/modified text).
• ~~associate word with document id (for cleanup)~~
• find more optimal ways to extract words and filter (may by use rayon and text indixing)

Please, check this out. Works without any lags on 10mb text file and small rust project loaded in buffers If it right way then I will try to make this PR more suitable

[estin]

any way i'm continued to implementing this feature and now it works like that

sequence diagram

Words associated with doc and doc-line on two separate hashmaps

• Doc-Words hashmap need to store initial doc's words, populated on load/save document
• Doc-Line-Words not accurate hashmap to store changed lines, populated on text changed and reseted on load/save docuemnt.

The Doc-Line-Words allow to do not accurate syncing with changed and unsaved text. It can contain duplicated data especially on lines deletion. But it's allow without any lags sends changed lines to the worker to update completion dictionary state.

PR is not perfect and may be improved by this tasks

• use BTreeSet instead of HashSet and try improve search speed by prefix (currently on HashSet works without lags)
• use oneshot signals pool
• ~~don't send doc text to the worker on :wq~~
• implement integration tests with language server
• get timeout value and completion items from config
• abstract completion item to separate option from language server and worker, it's allow implement ordering (ls items first) and some visualization

imho PR ready to review

[IvanGoncharov]

@estin Thanks a lot for this PR 🙇 For me, it is impossible to use helix as an editor for git commit messages without this feature (no completion from commit diff). I running this PR locally, and it works as a charm 👍

[3h15]

@estin Thanks for this PR. Same problem here with elixir.

[IvanGoncharov]

@estin Not sure if it is problem in your PR specifically, but I notice a strange behavior with the . operation, and I can reproduce it on master with LSP completion. Steps:

1. change some text using c
2. change another selection with .
3. completion menu is stuck on a screen.

See recording here: https://asciinema.org/a/zeenKdKViSoTQiYrodanWIhez

[AceofSpades5757]

@estin Not sure if it is problem in your PR specifically, but I notice a strange behavior with the . operation, and I can reproduce it on master with LSP completion. Steps:

1. change some text using c
2. change another selection with .
3. completion menu is stuck on a screen.

See recording here: https://asciinema.org/a/zeenKdKViSoTQiYrodanWIhez

The text getting stuck happens in other circumstances too. ctrl+c will fix it. I think there might have been an issue tracking it.

[estin]

@IvanGoncharov I hope completion menu is stuck now it fixed (see https://github.com/helix-editor/helix/pull/3328/commits/e3b9ed186619e4bbe1ab581721f5da68cd8f9bd9 ) You case of menu stuck was PR's bug

@AceofSpades5757 Can you show me another completion menu stuck? Currently on master at 2968756baf6208d48902607c8f8cb3c00c279149 I often saw this bug

• python
• insert class name Event() and wait on it
• opens popup with class constructor
• change cursor position to another line or switch buffer will't close it
• and as you say Ctrl-c will close it

Seems some bug on master, but I don't investigate time to research/solve it

[IvanGoncharov]

I hope completion menu is stuck now it fixed (see https://github.com/helix-editor/helix/commit/e3b9ed186619e4bbe1ab581721f5da68cd8f9bd9 ) You case of menu stuck was PR's bug

@estin Thanks, I can confirm it's fixed 😄

I also found another one, your completion doesn't honor completion-trigger-len in case I use default (which is 2), but now it triggers after the first key. Note: this sims to happen only on files without LSP completion (e.g. toml).

[estin]

@estin Thanks, I can confirm it's fixed

nice!

I also found another one, your completion doesn't honor completion-trigger-len in case I use default (which is 2), but now it triggers after the first key. Note: this sims to happen only on files without LSP completion (e.g. toml).

It's was planned like that.

There are two modes:

• with LSP: completion triggers as usual with honor for all current settings, and words completions would be merged to LSP result (two concurrent async requests to LSP and worker and then merge results)
• no LSP: on each char typed collects prefix and send request to the worker. It's very useful for me and works without any lags on huge files (tested on ~245Mb file json-line log)

P.S. I don't believe about that PR would be merge to upstream in near future. But it's allow me use helix as my primary editor, and I will sync it with master periodically for my personal usage )

[IvanGoncharov]

@estin Thanks for the explanation. In case the situation changes and this PR merges to master I think it should honor completion-trigger-len completion on the first char creates unexpected visual noise for me.

For now, I'm totally happy with PR, and big thanks for keeping it up to date 👍

[IvanGoncharov]

Thanks @estin 👍 With completion-trigger-len it works like a charm!

[nluetts]

I tested this feature with a couple of markdown files and had no issues. Would be great if this (or something similar) would find its way into helix.

[IvanGoncharov]

@estin Big thanks for keeping this PR up to date for almost a year 🙇 I use it daily, saving me a ton of time writing git commits messages. When writing a detailed description of my changes, I don't need to waste time and manually copy-paste identifiers from diff.

[danillos]

@estin great work. I can't see I'm using Helix without it anymore.

[estin]

Currently this PR is HACK, and marked as draft again. It works. (integration test would be failed - language server required)

• completion item required language_server_id
• to enable words completion document must has at least one language server
• to enable language server document must has url (file path)

To resolve this changes currently introduced in languages.toml

• stub languages server with command: rust-analyzer
• stub language with stub ls

May be this task must be solved in language server terms https://github.com/helix-editor/helix/discussions/6731

To enable words comletion on document without language server

1. Configure languages.toml

[language-server.stub]
name = "stub"
command = "<binary of any available language server>" # by default rust-analyzer

2. Set language for document (documen must has filepath)

:set-language stub

Future plans on this PR

• find simple way to implement embeded stub language server
• or allow completions without language server at all
• or waiting before core team resolve this task )

[pascalkuthe]

This feature should be possible to implement efficiently by running aho-corsik on the current buffer (as an async completion task, the infrastructure is already provided by the multi lsp PR).

aho-corsic already implements a streaming matcher (so works efficiently with ropey) and an (ASCII) ignore case option. Simply collecting all matches into a HashSet and expanding them to entire words should work quite well.

For performance reasons I dont think we want to run completions for every buffer tough. Instead the completion should simply scan the current buffer which is what you want in most cases anyway.

[estin]

Multi lsp feature allow to use external language server for completion. No longer needed to continue maintain this hacky PR.

For my use cases works fine with https://github.com/estin/simple-completion-language-server

By https://github.com/helix-editor/helix/pull/3328#issuecomment-1559031060 I will try implement naive not optimized language server

• incremental doc updates (use it's own ropey::Rope for each opened doc)
• on each completion request applied aho_corasick searcher on each opened doc as stream

[danillos]

Nice work @estin and thank you for work on it.