Providing a Language Server (LSP)

[mickael-menu]

zk can provide a basic integration with any LSP-compatible text editor by shipping a Language Server.

PR #21 implements a proof of concept server showing promising results, but there's more to come:

• Links

  • [x] Auto-complete internal Markdown links with [[
  • [x] Auto-complete the path portion of a Markdown link with [custom title]((
  • [x] User configuration to specify the link format using a template
  • [x] Open a note from an inline Markdown link or WikiLink (document link when supported and go to definition as an alternative)
  • [x] Open a website from an inline external Markdown link (with textDocument/documentLink)
  • [x] Open a reference link
  • [ ] Move the caret to the next link in the note with a code action, for quick navigation (with window.showDocument)
  • [x] Preview the content of a linked note with a hover
  • [x] Diagnostic (error) to show dead links
  • [x] Diagnostic (hint) to show the title of a linked note next to the link
  • [x] Browse backlinks with find references
  • [ ] Browse backlinks / outbound links with a tree-like call hierarchy (client support is still lacking)
  • [ ] Highlight WikiLinks using semantic tokens (client support is still lacking)
  • [ ] (Maybe?) Report DocumentLink for unlinked mentions, using aliases metadata

• Tags

  • [x] Auto-complete tags with # and : trigger characters
  • [ ] Rename a tag with a refactor code action
  • [ ] Highlight tags using semantic tokens (client support is still lacking)
  • [ ] View the list of all the notes for a particular tag when running the References action.

• Notes

  • [x] Code action to create (and link) a new note using the current selection as its title
  • [ ] Code action to create (and link) a new note using the current selection as its content (refactor.extract?)
  • [ ] Code action to rename the note title
  • [ ] User configuration to declare dynamic code actions to generate notes with custom group / template
  • [ ] Snippet support using zk templates

• Expose zk commands as LSP commands, for easy client-side consumption

  • [ ] zk.init to create a new notebook
  • [x] zk.index to index a notebook manually
  • [x] zk.new to create a new note
  • [x] zk.list to search for notes (returning a JSON)
  • [x] zk.tag.list to return the list of tags
  • [ ] zk.info to provide detailed JSON metadata about a notebook (list of groups, templates, dirs, etc.)

• Other

  • [ ] Optimizations for large (10k+) notebooks (esp. completion)

Feel free to share more ideas!

[mhanberg]

Tested this out, it's very cool!

I am seeing an issue using go to definition with neovim builtin lsp, but i'm not sure if that's a neovim problem or a zk problem. The error is Error executing vim.schedule lua callback: ...stalls/neovim/nightly/share/nvim/runtime/lua/vim/uri.lua:116: attempt to call method 'match' (a nil value)

Log

2021/04/04 21:00:24.323 DEBUG [zk.rpc] jsonrpc2: 

[mickael-menu]

@mhanberg It was actually caused by a bug in a third-party dependency. I opened a PR but forgot to disable the faulty portion in zk. You can try again from the latest main or the following pre-built binaries:

• zk-v0.3.0-4-g19f6009-macos-arm64.zip
• zk-v0.3.0-4-g19f6009-macos-x86_64.zip
• zk-v0.3.0-4-g19f6009-linux-amd64.tar.gz
• zk-v0.3.0-4-g19f6009-linux-arm64.tar.gz
• zk-v0.3.0-4-g19f6009-linux-i386.tar.gz

Btw, I tried the builtin LSP but couldn't make the trigger characters work for the completion. Did you setup anything special for this?

[mhanberg]

I updated and go to def works now 👍.

Btw, I tried the builtin LSP but couldn't make the trigger characters work for the completion. Did you setup anything special for this?

I use nvim-compe for completion.

[megalithic]

@mickael-menu first off, this is so great!

I've been using it with nvim-compe along with @mhanberg, too.

A couple items I've noticed.

1. When attempting to initiate auto-complete of files with [[ or [ depending on the mode you want, it doesn't complete the file extension, which then breaks go to definition, as soon as I manually add the the file extension, go to definition works as expected.
2. This is more of a question/confirmation of expected behaviour (the above might also be that, too); when auto-completing tags, should the \ escaping of space be converted back to ?

Other than those two minor items, this has been fantastic to use!

[mickael-menu]

@megalithic

1. When attempting to initiate auto-complete of files with [[ or [ depending on the mode you want, it doesn't complete the file extension, which then breaks go to definition, as soon as I manually add the the file extension, go to definition works as expected.

I set up the links for my own usage and I usually prefer to omit the extension. But I'll let users configure the format of the generated links in the config later on.

However, the link href is matched as a prefix of the actual path, so it should work without the extension or if you have a unique ID prefix. For example, [[202005201056]] would match 202005201056 Interesting subject.md. Could you share an example of link + note path pair which doesn't work? And what do you see in the LSP logs?

2. This is more of a question/confirmation of expected behaviour (the above might also be that, too); when auto-completing tags, should the \ escaping of space be converted back to ?

This is expected, because a normal #hashtag can't contain spaces, so I added this ad-hoc escape syntax if an existing tag contains spaces. But if you enable Bear's multi-word tags syntax in the config, instead of using \ escape character the completed tag will be like #multi-word tag# What tag syntax did you use so far supporting spaces?

[megalithic]

Thanks @mickael-menu .. I'm still playing around with this to get a repro case for you. I think I might be using something wrong; meaning, I'm looking to be able to [[ then rely on "friendly" titles to complete on. I've also noticed that not all items in my main notebook are showing up in the completion menu. I'll get a loom or something going to demonstrate today.

[megalithic]

Ahh, I'm wondering if certain characters in a filename or title might be breaking completion? |, :, and & are command items in some of my note file names and titles (i auto-extract calendar titles and use those from meeting note generation and pass that along to zk.nvim to generate the note, or open an existing note with the same file name).

Will check the logs to verify this.

When searching for a note (using zk edit --interactive via cli, not abstracted out via zk.nvim) that does contain one of the above characters, these are throw into the preview area of fzf:

[mickael-menu]

I think I might be using something wrong; meaning, I'm looking to be able to [[ then rely on "friendly" titles to complete on

Yes right now the completion is on the note title (heading or YAML frontmatter) and not on the file path. I intend to add both in the completion filter later.

I'll open a new issue for the special characters.

[mhanberg]

Go to definition seems to be working when the current note is in the root of the notebook, but not when the note is in a sub-directory. I noticed this when implementing the "daily journal" example from the docs.

[mickael-menu]

@mhanberg Thanks for spotting that, should be fixed in main

zk-v0.3.0-5-gdd561be-macos-arm64.zip

[mickael-menu]

I merged in a way to specify the completion link format from the user configuration: https://github.com/mickael-menu/zk/pull/32

[format.markdown]
# Format used to generate links between notes.
# Either "wiki", "markdown" or a custom template. Default is "markdown".
link-format = "wiki"
# Indicates whether a link's path will be percent-encoded.
link-encode-path = false
# Indicates whether a link's path file extension will be removed.
link-drop-extension = true

Incidentally, internal link completion is triggered only with [[ now, not a single [. You need to setup your preferred link syntax in the zk config.

[megalithic]

@mickael-menu .. one thing i've noticed recently (not sure if it's a zk lsp implementation or related to neovim's lspconfig and native LS client implementation), is that when trying to "select" a completion item that has spaces it will break the selection and you'll end up with just First\ instead of First\ Word or First Word. Is this something that a.) works on coc.nvim; and b.) works for you with nvim-lspconfig and nvim's native lsp client implementation, that you've seen?

Thanks!

[megalithic]

Oh man, update; turns out bullets.vim was the culprit :( that's a bummer.

Soooo.. please ignore my above question. <CR> selection on an autocomplete item that has spaces, works just fine. :)

[mickael-menu]

Ha that's interesting because I actually had the same issue with coc.nvim, which was considering spaces as the end of the completion item for some reason. I had to set this in its settings to make it work:

{
  // Important, otherwise link completion containing spaces and other special characters won't work.
  "suggest.invalidInsertCharacters": []
}

[mickael-menu]

I created a Visual Studio Code client extension for zk, available at https://github.com/mickael-menu/zk-vscode

[ghost]

In case anyone else gets stuck when using Pandoc in Vim, just add get_language_id = function() return 'markdown' end to your config (for Neovim's built-in LSP). This forces the filetype that get passed to the LSP as markdown (otherwise it will send markdown.pandoc) so that the LSP can work.

E.g:

configs.zkls = {
  default_config = {
     cmd = {'zk', 'lsp', '--log', '/tmp/zk-lsp.log'},
     filetypes = {'markdown', 'markdown.pandoc'},
     get_language_id = function() return 'markdown' end,  -- force language ID to markdown so that LS will work
     root_dir = function() return vim.loop.cwd() end,
     settings = {},
  };
}

[ghost]

On a related note, does filtering the tag/note auto-completion work? If I enter the trigger character (# or [[), I get a popup with all my tags/notes, but as soon as I start typing to "filter" through the list, I just get normal (non-LSP) auto-completion. I'm using the built-in Neovim LSP client and nvim-compe, so not sure if it's an issue with one of them?

[mickael-menu]

@ZachariasLenz Thanks, I'll add this to the documentation!

On a related note, does filtering the tag/note auto-completion work?

It depends what you expect but with coc.nvim I can filter though the auto-completion list with fuzzy-matching. It matches the notes' title and their path (which is hidden from the completion items).

as soon as I start typing to "filter" through the list, I just get normal (non-LSP) auto-completion

I'm not sure what you mean by non-LSP auto-completion? Would you be able to record a screencast?

[ghost]

@mickael-menu see the screencast for an example of what I mean:

https://user-images.githubusercontent.com/81169491/116593769-fd597000-a910-11eb-8c80-f9fd4d892c9c.mp4

[mickael-menu]

@ZachariasLenz Thanks, this definitely doesn't look right. I'm not sure what could be the problem as I'm not using NeoVim's built-in LSP yet. Do you have any idea @megalithic?

(Nice color scheme ;))

Here's how it looks like with coc.nvim

https://user-images.githubusercontent.com/58686775/116608753-dc564680-a933-11eb-9977-116022ac27f9.mov

[mickael-menu]

I just merged some pretty exciting new LSP features.

I added two code actions to create a new note using the text selection as title. Either in the current directory, or the root/top directory. It will convert the selection to a link to the newly created note.

This feature builds on new custom LSP commands: zk.index and zk.new. Using your editor's API, you can directly call these commands which can be used to create keybindings around various note creation use cases. Here's a screencast:

https://user-images.githubusercontent.com/58686775/117876567-cdf91a80-b2a3-11eb-98c5-b5f07aaed16f.mov

You will need a bit of configuration to add user commands and shortcuts exposing zk.index and zk.new.

This is my coc.nvim config:

~/.config/nvim/init.vim

command! -nargs=0 ZkIndex :call CocAction("runCommand", "zk.index", expand("%:p"))
command! -nargs=? ZkNew :exec "edit ".CocAction("runCommand", "zk.new", expand("%:p"), <args>).path

nnoremap <leader>zi :ZkIndex<CR>
nnoremap <leader>zn :ZkNew {"title": input("Title: ")}<CR>
nnoremap <leader>zl :ZkNew {"dir": "log"}<CR>

(Note the cool keybinding which prompts for a title: nnoremap <leader>zn :ZkNew {"title": input("Title: ")}<CR>)

And here's with built-in Neovim LSP client:

~/.config/nvim/init.vim

command! -nargs=0 ZkIndex :lua require'lspconfig'.zk.index()
command! -nargs=? ZkNew :lua require'lspconfig'.zk.new(<args>)

lua << EOF

local lspconfig = require'lspconfig'
local configs = require'lspconfig/configs'

configs.zk = {
  default_config = {
    cmd = {'zk', 'lsp'};
    filetypes = {'markdown'};
    root_dir = lspconfig.util.root_pattern('.zk');
    settings = {};
  };
}

configs.zk.index = function()
  vim.lsp.buf.execute_command({
    command = "zk.index",
    arguments = {vim.api.nvim_buf_get_name(0)},
  })
end

configs.zk.new = function(...)
  vim.lsp.buf_request(0, 'workspace/executeCommand',
    {
        command = "zk.new",
        arguments = {
            vim.api.nvim_buf_get_name(0),
            ...
        },
    },
    function(_, _, result)
      if not (result and result.path) then return end
      vim.cmd("edit " .. result.path)
    end
  )
end

lspconfig.zk.setup({
  on_attach = function(client, bufnr)
    -- Key mappings
    local function buf_set_keymap(...) vim.api.nvim_buf_set_keymap(bufnr, ...) end
    local opts = { noremap=true, silent=true }
    buf_set_keymap("n", "<CR>", "<cmd>lua vim.lsp.buf.definition()<CR>", opts)
    buf_set_keymap("n", "K", "<cmd>lua vim.lsp.buf.hover()<CR>", opts)
    buf_set_keymap("n", "<leader>zi", ":ZkIndex<CR>", opts)
    buf_set_keymap("v", "<leader>zn", ":'<,'>lua vim.lsp.buf.range_code_action()<CR>", opts)
    buf_set_keymap("n", "<leader>zn", ":ZkNew {title = vim.fn.input('Title: ')}<CR>", opts)
    buf_set_keymap("n", "<leader>zl", ":ZkNew {dir = 'log'}<CR>", opts)
  end
})
EOF

[mickael-menu]

zk now reports diagnostics for dead links and wiki-link titles.

The wiki-link titles are particularly useful with Neovim's 0.5 built-in LSP client, as they are displayed as virtual text. However I disabled this diagnostic by default as it's not so useful for other editors. You need to enable it in your zk config:

[lsp.diagnostics]
# Report titles of wiki-links as hints.
wiki-title = "hint"

Here's a screenshot after customizing the colors:

~/.config/nvim/init.vim

highlight LspDiagnosticsDefaultError ctermfg=red guifg=red
highlight LspDiagnosticsUnderlineError ctermfg=red guifg=red
highlight LspDiagnosticsDefaultHint ctermfg=yellow guifg=yellow
highlight LspDiagnosticsUnderlineHint cterm=none gui=none

• zk-v0.4.0-5-gaa68199-macos-x86_64.zip
• zk-v0.4.0-5-gaa68199-linux-i386.tar.gz
• zk-v0.4.0-5-gaa68199-linux-arm64.tar.gz
• zk-v0.4.0-5-gaa68199-linux-amd64.tar.gz

[jrekier]

(Note the cool keybinding which prompts for a title: nnoremap <leader>zn :ZkNew {"title": input("Title: ")}<CR>)

I'm able to get the zn command to work, including the prompt for a title. However, there appears to be no sign of the entered title once the note is created (see recording below)

https://user-images.githubusercontent.com/9407736/118467264-f352b200-b703-11eb-9774-78f933009704.mov

I suspect I might be doing something wrong ...

[mickael-menu]

@jrekier Looks like your template is empty. In my zk notebook I have:

• .zk/config.toml

  [note]
  template = "default.md"
  

• .zk/templates/default.md

  # {{title}}
  
  {{content}}
  

You can find the available placeholders for the templates in the documentation.

[jrekier]

indeed ... thanks !

Now I understand the use of templates better. Changed mine to use yml front matter:

• .zk/templates/default.md

---
title: {{title}}
---
{{content}}

and it works wonders. Cheers

[mickael-menu]

Btw I found out that we can display diagnostics as virtual text with coc.nvim too, if you don't want to upgrade to Neovim 0.5.

You will need this in :CocConfig:

  "diagnostic.virtualText": true,
  "diagnostic.virtualTextCurrentLineOnly": false

Also I suggest using quotes around your title if you use a YAML front matter, as special characters might throw off the parser.

[entropitor]

Will there be auto completion for full markdown links as well? [ should trigger auto complete for markdown links with the canonical title as text and path filled in correctly as url

It also looks like [[ links get auto-completed with a third [ is this on purpose? And is there any way to autocomplete the filename instead of the note title?

And why do we need two ( (](() to trigger markdown path completion? The path completion itself also doesn't seem to work for me. I get all the options but when I select one, nothing happens.

(I'm working in neovim so it could be editor specific)

[mickael-menu]

Will there be auto completion for full markdown links as well? [ should trigger auto complete for markdown links with the canonical title as text and path filled in correctly as url

Yes it's available already, but it is still triggered by [[ as I think it would be undesirable to have the auto-completion activated for any normal Markdown link (including external links, image embedding, etc.). You need to customize the link format generated in your config file.

[format.markdown]
link-format = "markdown"
link-drop-extension = true
link-encode-path = true

However you can have only a single link format completed with [[, so it wouldn't be very useful if you are mixing Wiki-links and regular Markdown links, for internal links. Is it something you need? Can you explain more your usage if it's the case?

It also looks like [[ links get auto-completed with a third [ is this on purpose?

No, but if you are using completion-nvim, I noticed it is not applying LSP's additionalTextEdits despite claiming it which is supposed to remove the extra [[. Maybe I missed something in the config. I actually have a lead that I could explore for this but it requires Neovim-specific LSP handler. Until then, nvim-compe or coc.nvim should work properly.

And is there any way to autocomplete the filename instead of the note title?

I'm adding the path to LSP's filterText which should let the editor filter-complete with the path too, even if it is not visible in the completion pop-up. However again this is highly dependent on your editor/plugins. Coc.nvim has been the most reliable for me as it implements a lot of the LSP spec.

However, I could see how it would be interesting to be able to customize the label of the completion pop-up. I'll think about it.

And why do we need two ( (](() to trigger markdown path completion? The path completion itself also doesn't seem to work for me.

This is actually an alternative to [[. Personally I only use regular markdown links and no wiki-links. Sometime I want to link to a note using its title, so I'll call [[. But often I want to customize the title of the link to put it in context of the current note. In this case I will type [custom title](( and this will complete the path to other notes without replacing my custom title.

I used (( for the same reason as using [[ instead of a single [: prevent triggering the completion when I don't want to link to an internal note.

I get all the options but when I select one, nothing happens.

Probably an editor/config issue, I noticed this issue sometime with conflicting plugins in Neovim.

[entropitor]

Okay, I didn't know about these settings! And with nvim-compe it seems to work much better indeed.

I understand the philosophy better now and while I don't fully agree, I can live with it and it kinda makes sense 👍

Thanks for the extraordinary detailed help!

[danymat]

Hello, I'm the creator of a basic lsp for my Zettelkasten purposes : https://github.com/lsp-zettelkasten/lsp-zettelkasten It seems we have a lot to share in common (hello @mickael-menu 😃 )!

I initially wanted a fully independent LSP for managing my notes. As I switched from obsidian to Neovim, I lost some good stuff such as automatic linking completion, tags management, etc.

I created the LSP aiming to allow the user to use it without installing something more than the server

I did managed to create the option to create links on the fly, by typing [[.

I'm now stuck at creating a tree-sitter parser for recognizing the links and tags, allowing me to propose tags based on current ones, and find links more quickly.

I love the idea of this lsp, which is leveraging the zk utility. Nevertheless, I would love to see a fully independent lsp emerge, from your lsp implementation (👋🏻 @mickael-menu ), mine, or a merge from both.

I'm open at helping and receiving help, because I'm convinced that what matters most is a centralized and participative implementation for the good of us, knowledge workers.

I have some ideas about leveraging tree-sitter in order to handle tags, highlighting, for our zettelkastens. Feel free to contact me at [email protected] or here below.

Sincerely, Daniel