[plugin] org-roam.nvim progress & discussion

[chipsenkbeil]

About this task

This issue started discussing a potential bug, and ended up evolving into a rundown of the ongoing work on org-roam.nvim, a plugin to implement Org-roam in neovim.

Scroll further down to see details about the plugin, code pointers and highlights, etc.

The original bug report is kept below for clarity.

Describe the bug

I am writing a plugin that creates some buffers with a buftype of nofile. The purpose of these is to generate some immutable orgmode content that you can navigate. In particular, I want to take advantage of the fantastic folding that orgmode offers.

Unfortunately, when I go to collapse a section, I get an error about the current file is not found or not an org file, caused by https://github.com/nvim-orgmode/orgmode/blob/261c987345131a736066c25ea409f4d10904b0af/lua/orgmode/files/init.lua#L106

Steps to reproduce

1. Create a scratch buffer via vim.api.nvim_create_buf(false, true)
2. Set the filetype to org via vim.api.nvim_buf_set_option(bufnr, "filetype", "org")
3. Populate with some org headings via vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, {"* Heading 1-a", "** Heading 2", "* Heading 1-b"})
4. Navigate to the buffer and hit tab to try to collapse or expand a section

Expected behavior

Section is properly folded.

Emacs functionality

No response

Minimal init.lua

-- Import orgmode using the minimal init example

Screenshots and recordings

No response

OS / Distro

MacOS 14.4

Neovim version/commit

0.9.5

Additional context

Using orgmode on master branch at commit 651078a.

[kristijanhusak]

File is considered valid only if it has an org or org_archive extension. Try setting a name for a buffer that has an org extension with nvim_buf_set_name

  local bufnr = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_option(bufnr, 'filetype', 'org')
  vim.api.nvim_buf_set_name(bufnr, 'somename.org')
  vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, { '* Heading 1-a', '** Heading 2', '* Heading 1-b' })
  vim.cmd('b'..bufnr)

[chipsenkbeil]

@kristijanhusak that solved the error about not being a current file! I'm hitting an issue where the plugin still doesn't detect a fold. There may be something wrong with my buffer or setup. Applies on any heading.

[kristijanhusak]

What would you expect to be folded here? Did you try doing zx to recalculate folds? I see you are trying to implement org-roam. I had similar idea but didn't catch time to start working on it.

[chipsenkbeil]

The first heading. If I create a file manually and reproduce the contents, I can fold it.

I had to make the repo private while I got permission to work on it on personal time. Now that I have it, here's the current plugin: https://github.com/chipsenkbeil/org-roam.nvim

About the plugin

Database

I wrote a simplistic implementation of a database with indexing supported that provides an easy search for links and backlinks. It isn't as full-fledged as SQL and I'm sure will struggle with larger roam collections, but for me this is enough.

Parser

Leveraging the orgmode treesitter parser for me to find the details I need to build the above database.

Completion of node under cursor

Covers both completing within a link and under a cursor. Essentially does the complete everywhere out of the box. Here's a demo:

https://github.com/nvim-orgmode/orgmode/assets/2481802/0c71540d-fdca-47e3-b0b4-74c61e56ae93

Quickfix jump to backlinks

I like the quickfix list, so while Emacs doesn't have this and uses an org-roam buffer, this was easy to whip up for neovim:

https://github.com/nvim-orgmode/orgmode/assets/2481802/a4ece5f8-c534-4a0e-9e03-f386a501e1ab

Org buffer

I was generating an org file instead of writing a custom buffer. I may switch to the custom buffer highlighting and format anyway because I need to capture link navigation to open in a different buffer versus the current one.

Id navigation

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently. So I'll need to write an org-roam variant to support opening id links. Shouldn't be hard at all. I also built out id generation for a uuid-v4 format in pure Lua. My test nodes aren't using it, though.

[kristijanhusak]

Thanks for opening it up. I just skimmed through it, and it seems you did a lot of custom stuff. A recent refactor was also done to support org-roam with some of the internals. This is what I had in mind:

1. Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51
2. Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though
3. Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23
4. Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently

In the issue description you wrote you are using commit 651078a2fe60b12c93903e3a2b655491c951bf9d. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

[chipsenkbeil]

Thanks for the pointers! When I looked at the plugin's API and data structures, it was (and I believe still is) missing crucial information I'd need for a fully-functional org-roam implementation. Would it make sense to move this discussion to a separate issue? I know you have the plugin API pinned issue, but there's a lot of different questions and dialog I'd want to have about needs and usage that feels better for a separate issue.

Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under cursor, etc.

Wonder how much of that has changed or was misinformed from my first look.

I also didn't check to see how you're loading files. I tried to keep most of my logic async - in the sense of leveraging neovim's scheduler and callbacks - so I can (in the future) easily report loading a large set of files, monitoring file changes and reloading, etc.

Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though

I don't know on this one. The database I wrote is a way to both track outgoing links (ones you'd find in a file) and incoming links (aka backlinks into a file). I like the design I've got, so I'll most likely keep this.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

In the issue description you wrote you are using commit https://github.com/nvim-orgmode/orgmode/commit/651078a2fe60b12c93903e3a2b655491c951bf9d. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Good to know. I've got different versions on different machines. Given I started fiddling with this a couple of months back, I guess the org id was implemented after.

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

I want to reduce the hacks and redundancy where possible. My implementation is meant to build on top of your plugin to supply the org-roam features, but when I started it looked like there was enough missing that I ended up only leveraging the treesitter language tree to get the information I needed.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

[kristijanhusak]

We can either have a separate issue or rename this one.

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under the cursor, etc.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using luv. Most methods in file and headline provide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

[chipsenkbeil]

We can either have a separate issue or rename this one.

Renamed this one.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using luv. Most methods in file and headline provide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

I'll give these a look in a week or two to see how they operate.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I don't know if it makes sense to add it to that method, but the locations of links within a file are needed for org-roam in order to support listing multiple references to the same backlink and to be able to pull in a sample of the content that linked to a node. I think it's easier to see from this person's tutorial of the Emacs usage: https://youtu.be/AyhPmypHDEw?si=mLGsAdosnKjTZ-1f&t=1690

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

I was really hoping that I could reuse it, so this makes me happy to hear. :) Will be giving that a look as soon as I reach that point.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Got it. Yes, I definitely want to use your code to handle omnicompletion. The selector is more like a built-in telescope interface, which I wanted to have similar to what is shown in the emacs tutorial above when it comes to selecting nodes where you can not only select between choices but also filter the choices further. So having both is my plan.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

I don't really mind where my plugin lives, so if you would be interested in taking this in at some point in the future once we remove as much of the custom logic as makes sense, then I'd be happy to hand it over to you. I need something like this for work, and it didn't exist, which is why I'm building it now.

[kristijanhusak]

I added range property to links in c4eeb3d9caa6403583e6d2285627126d70aef691 that holds the position of the links, including [[ and ]] markers.

We can add a few more things as we go if you find them missing, just let me know.

[chipsenkbeil]

@kristijanhusak are these indexed starting at 0 or 1?

[kristijanhusak]

It's 1.

[chipsenkbeil]

@kristijanhusak I just updated to c4eeb3d and when trying to open an id link using org_open_at_point (bound to a local leader mapping), I'm still getting the issue about "No headline found with id: ...". I'm assuming this is because having a file-level id is unique to org-roam and not orgmode, which I "think" only has the concept of ids in property drawers at the heading level. But I'm not sure if global ids follow that logic or not.

From https://orgmode.org/manual/Handling-Links.html

If the headline has a ‘CUSTOM_ID’ property, store a link to this custom ID. In addition or alternatively, depending on the value of org-id-link-to-org-use-id, create and/or use a globally unique ‘ID’ property for the link 28. So using this command in Org buffers potentially creates two links: a human-readable link from the custom ID, and one that is globally unique and works even if the entry is moved from file to file. The ‘ID’ property can be either a UUID (default) or a timestamp, depending on org-id-method. Later, when inserting the link, you need to decide which one to use.

Given this fact and your desire to maintain only core orgmode functionality within this plugin, I'm assuming that unless the global ID at a file level is part of core orgmode, I will need to write my own open logic for links that navigates to the correct id that supports file-level ids. The thought process is to have that function fall back to your implementation if the link is not an id link. Thoughts?

[chipsenkbeil]

I changed the org-roam buffer to more closely match the org-roam variation. Looks like they changed to use magit underneath, which I'm just replicating in style right now.

Also, progress on following node change under cursor. Don't know the performance considerations of this given I've got tiny tests.

https://github.com/nvim-orgmode/orgmode/assets/2481802/c454bcb6-15a7-41cf-893f-fed9a752d1b0

[kristijanhusak]

Id links work fine for me. There's also a test that confirms that it's working. Note that your files need to be part of org_agenda_files otherwise, it won't be found.

[chipsenkbeil]

@kristijanhusak your test has a property drawer with an id that is within a section with a headline. Does it work with a top-level property drawer?

The case I'm referring to is a top-level property drawer not within a section.

My dotfiles configure every org file (including those of the roam directory) to be within the agenda:

[kristijanhusak]

Ah yes, you are correct, there was no support for top-level id. I added that now on the latest master, let me know if it's not working as expected.

[chipsenkbeil]

@kristijanhusak fantastic! Quick test has it working just fine. :) One less thing I have to manage myself.

[chipsenkbeil]

@kristijanhusak I haven't been able to find it yet. Do you have a version of org-id-get-create (part of org-id.el)?

This is referenced in org-roam's manual and is used to inject an ID into a head as seen in this demo.

[kristijanhusak]

Yes, you can use this https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

local id = require('orgmode.org.id').new()

There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.

[chipsenkbeil]

There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.

This is what I'm specifically looking for, which both does the work of generating the id and placing it in the headline. Is there a version that also works with a property drawer at the file level?

When I tested org-id-get-create in Emacs, it works with a property drawer at the file level. So, ideally, this would be a command that someone could run whether they're in a heading or not.

Here's an example:

https://github.com/nvim-orgmode/orgmode/assets/2481802/bf97e15f-2a27-4ed4-bf7b-81f06a9a2366

[kristijanhusak]

We could add that, but I don't think you will need it. For org-roam you will have custom templates that will already populate this information before showing the capture window. When creating a template, just call the orgmode.org.id to generate an id for you.

function OrgRoamTemplate:compile()
   --call parent compile
  local content = OrgTemplate:compile()
  content =  vim.list_extend({
    ':PROPERTIES:',
    ':ID: '..require('orgmode.org.id').new(),
    ':END:'
  }, content)
end

[chipsenkbeil]

@kristijanhusak I'm switching back to having the org-roam buffer be an actual org file w/ syntax. I've seen examples of org-roam that use magit as the interface and others where it has a plain org file.

From the discussions I've seen regarding how the buffer is used, people will either use it to open a backlink/citation/unlinked reference directly in the frame (via RET) or they can force it to open in a different frame (via C-u RET).

I know OrgMappings:open_at_point() exists to open the link or date at the point given. Is there any function that lets you open at point while specifying a different window? Was looking at that function and considering if I need to build a wrapper for that function to be able to point to a different window.

Org-roam manual reference

Link: https://www.orgroam.com/manual.html#Configuring-the-Org_002droam-buffer-display

Crucially, the window is a regular window (not a side-window), and this allows for predictable navigation:

• RET navigates to thing-at-point in the current window, replacing the Org-roam buffer.
• C-u RET navigates to thing-at-point in the other window.

Emacs manual reference

Link: https://orgmode.org/org.html#Handling-Links

There's a minor reference to org-link-frame-setup, which appears to let you configure how files and directories are opened including other frames:

There's a minor reference to org-link-use-indirect-buffer-for-internals, which I opened up below:

Other references

• https://stackoverflow.com/questions/8881649/how-to-force-org-mode-to-open-a-link-in-another-frame

[kristijanhusak]

We could extend it to accept the command that opens the "point", and default it to edit. Would that work for you?

[chipsenkbeil]

We could extend it to accept the command that opens the "point", and default it to edit. Would that work for you?

I think that should work, yeah. As long as there is a way for me to specify the window to use versus the current window.

I think I'd do this by using wincmd:

---@param winnr integer # id of window where we will open buffer
---@return string # to supply to vim.cmd(...)
function make_cmd(winnr)
    local cmd = winnr .. "wincmd w" -- goes to window "winnr"
    return cmd .. " | edit"
end

Update

Leveraging loading a singular orgfile per preview in the org buffer. I was reading through how org-roam does this (via changelog and source), and it looks like it handles certain cases specially such as detecting if the link is in a heading and just showing the heading text or detecting if the link is in a list item and showing the entire list.

I'll be switching this back over to an org buffer soon, but here's a preview:

This is my logic for now, although I'm assuming I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version? In my old approach, I would stat each file to check its mtime.sec against a cached version.

-- NOTE: Loading a file cannot be done within the results of a stat,
--       so we need to schedule followup work.
vim.schedule(function()
    require("orgmode.files")
        :new({ paths = opts.path })
        :load_file(opts.path)
        :next(function(file)
            ---@cast file OrgFile
            -- Figure out where we are located as there are several situations
            -- where we load content differently to preview:
            --
            -- 1. If we are in a list, we return the entire list (list)
            -- 2. If we are in a heading, we return the heading's text (item)
            -- 3. If we are in a paragraph, we return the entire paragraph (paragraph)
            -- 4. If we are in a drawer, we return the entire drawer (drawer)
            -- 5. If we are in a property drawer, we return the entire drawer (property_drawer)
            -- 5. If we are in a table, we return the entire table (table)
            -- 5. Otherwise, just return the line where we are
            local node = file:get_node_at_cursor({ opts.row, opts.col - 1 })
            local container_types = {
                "paragraph", "list", "item", "table", "drawer", "property_drawer",
            }
            while node and not vim.tbl_contains(container_types, node:type()) do
                node = node:parent()

                -- Check if we're actually in a list item and advance up out of paragraph
                if node:type() == "paragraph" and node:parent():type() == "listitem" then
                    node = node:parent()
                end
            end

            -- Load the text and split it by line
            local text = file:get_node_text(node)
            item.lines = vim.split(text, "\n", { plain = true })
            item.queued = false

            -- Schedule a re-render at this point
            opts.emitter:emit(EVENTS.CACHE_UPDATED)
            return file
        end)
end)

[kristijanhusak]

we could also make the argument a function, that would default to:

local edit_cmd = edit_cmd or function(file) return 'edit '..file end

vim.cmd(edit_cmd(filename))

That might give you more control over it.

I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version?

Yes, you should go with the single instance. Here I have an Org instance that holds other instances. Regarding caching, I do same for files that are not loaded, and check the buffer changedtick if the file is loaded inside a buffer. You can see the logic here https://github.com/nvim-orgmode/orgmode/blob/b7c42e6fc5982afef5e322b33fe58eec9a09c76d/lua/orgmode/files/file.lua?plain=1#L108-L119

[chipsenkbeil]

we could also make the argument a function, that would default to:

local edit_cmd = edit_cmd or function(file) return 'edit '..file end

vim.cmd(edit_cmd(filename))

That might give you more control over it.

I think either solution should work.

Org roam buffer

I'm still hitting some quirks with creating an org buffer that is of buftype = nofile.

1. As you pointed out earlier, I have to supply .org at the end of the buffer name. It looks like as a result of this, orgmode updates the buffer name to be relative to the orgfiles directory for me. So now if I want to ensure that I'm not within my own buffer, I have to do vim.endswith(vim.api.nvim_buf_get_name(0), "org-roam-buffer.org") instead of equality.
2. It looks like my buffer's preferences to be both unlisted and a scratch buffer get overwritten. Maybe you're swapping out the buffer on me (don't know yet). This means that this temporary buffer shows up in the buffer list.
3. I still can't get folding to work. Running zx doesn't appear to update the folds.
4. Reloading the buffer clears the contents (this is on me to fix) and requires to jump elsewhere to refresh. But the callout is that the syntax highlighting appears to break partially when the buffers contents are reloaded again. The comment and heading colors.

https://github.com/nvim-orgmode/orgmode/assets/2481802/04aa86b3-ed93-42c6-b065-9e8c822d0596

[kristijanhusak]

I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.

[chipsenkbeil]

I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.

Sure, I can experiment. I think the broader issue is it seems like orgmode really wants org buffers to represent files on disk, while I am making a buffer-only org document with content that I dynamically generate and update on cursor change.

[chipsenkbeil]

@kristijanhusak still no luck, so there must be something else I'm doing wrong. Even persisting the file to disk, the folds aren't working. Everything else is fine.

The logic is in the NodeView Window, where I'm maintaining an OrgFiles cache across all windows and another cache where I've calculated lines to show in a preview per backlink. All of that works fine, so I must have something wrong set up w/ creating the org buffer.

Maybe there's something else wrong in my setup. I have to hack orgmode a bit to get it to work, including a full reload (edit!) for the current buffer when the orgmode plugin first loads.

---

It does look like the ftplugin is triggering for org as I can see b:undo_ftplugin is set. For some reason, the fold levels aren't being detected. Everything has a foldlevel of 0.

Seems like if I change from being nomodifiable and nofile to a regular buffer and modifiable and then write the file, I can get folding. I also need to then reload the file via :e to get folds to fully work as some of them don't fold properly. When I took a look at your autocmds, it looks like you just do require("orgmode"):reload(file), which I tried to replicate with the name of the buffer, but so far I can't get it to work unless I make it modifiable, manually write via :w, and then reload via :e.

[kristijanhusak]

I investigated a bit what's happening, and I think just additionally doing filetype detect should solve an issue for you. Here's a fuction that I used for testing (simplest to be put in init.lua):

_G.test_org_buffer = function()
  local buf = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_option(buf, 'filetype', 'org')
  -- You can maybe use some naming convention like this, but not necessary
  vim.api.nvim_buf_set_name(buf, 'org:///org-roam-backlinks.org')
  vim.api.nvim_buf_set_lines(buf, 0, -1, false, { '* level 1', '** level 2', '** level 2.2' })
  vim.cmd('b'..buf)
  vim.cmd('filetype detect')
end

1. Open nvim
2. execute :lua test_org_buffer()