Create Shims for "Popular" File Explorer Plugins

[miversen33]

Once the shim manager is in place (See branch explorer), we will need to get some shims in place so that netman can self integrate with more popular explorers. Of these, the following shims likely need to be created

• [x] Netman.nvim of course ;)
• ~~[ ] nvim-tree~~ Not supporting currently. See this comment
• [ ] plenary.nvim
• [ ] neo-tree.nvim Per Request
• [ ] nerdtree
• [ ] chadtree

[miversen33]

Merging of PR 29 completes a simplistic Netman.nvim file manager and shim

[miversen33]

It looks like alot of this will be revolving around "reimplementing" libuv within netman, with specific hooks within the shim to redirect calls to providers if we (Netman) handle the provider details.

[miversen33]

nvim-tree is proving difficult as it utilizes a lower level vim command for managing its working directory (M.force_dirchange) as opposed to neovim provided lua functions. In particular, it uses the vim commands :lcd and :cd which are directly implemented in c

I am going to put a pin in shimming nvim-tree for the time being and focus on plenary (as this should provide direct support for telescope-file-browser, my preferred file browser)

[miversen33]

As I apparently dont know how to use Github, this is being worked on (along with #36) in branch issue-28-libuv-shenanigans

[miversen33]

Note, this branch currently has a working (albeit, with absolutely horrendous performance) interface with Plenary and thus telescope-file-browser

I still have some metadata work to do and likely the entire thing needs a massive refactor.

Additionally, it is worth calling out that there will be some minor changes to the API spec (because who wants stable API's anyway?).

Namely, read will now require returning the remote current working directory whatever you are returning. This will be documented more on release and merge into main.

[miversen33]

There are a few notable issues that still need addressing

• The names displayed for files are truly terrible (they are the absolute path as returned from our libuv mask). I am not sure the best way to deal with this as these names are the same names that are passed to us to open them via the API, so we need the names to be absolute. More digging will occur. Potentially insight could be provided via @fdschmidt93 but unless/until that happens, I will continue poking through the make_entry code in telescope-file-browser
• Performance is literally awful. There are some quick fixes that can be done to address this however, namely a stat cache that our libuv mask implements to help prevent the repetitive calls that Telescope, telescope-file-browser, and plenary do for paths. There is also some pretty garbage code powering the metadata parsing in the docker module that could stand for significant improvement. I will start by getting a cache targeted as that will be easier and show more immediate and drastic results for performance.
• Previewing remote resources currently doesn't work as that part of the libuv stack has not been implemented yet. Still a WIP

A fun note, this does handle hellish filenames, as shown in the above screenshot :) I believe I have the shell escaping working completely now

[miversen33]

Removing a lot of the debugging logs (and reducing the remaining to a level under debug), along side with implementing a rudimentary LRU cache has helped out with remote performance drastically, however there is still plenty of optimization that needs to be had within the API to help prevent unnecessary calls to providers for repetitive data (metadata for example).

Also need to optimize how we are processing stuff within the providers as it kinda really sucks right now :upside_down_face:

[fdschmidt93]

Potentially insight could be provided via @fdschmidt93 but unless/until that happens, I will continue poking through the make_entry code in telescope-file-browser

The entry maker could use some RFC, since it is one of the slowest component in the file browser atm. telescope-file-browser casts all results as plenary.paths (maybe currently unnecessarily so, I'll have to check).

The names displayed for files are truly terrible (they are the absolute path as returned from our libuv mask). I am not sure the best way to deal with this as these names are the same names that are passed to us to open them via the API, so we need the names to be absolute.

The entry.display is a either a string or a function that's called lazily. Of the very top of my head I suppose something goes quite wrong with processing the path (making it relative). Relevant code is here.

[miversen33]

The entry.display is a either a string or a function that's called lazily. Of the very top of my head I suppose something goes quite wrong with processing the path (making it relative). Relevant code is here.

This is correct. Unfortunately I dont think there is much that can be done to improve this as this relies on some deeply seeded logic in plenary's Path library to determine if it should be relative or not. Without going to in depth, telescope-file-browser reaches out to Plenary to get the relative path (via a chain off local path_display = utils.transform_path(opts, entry.path)) and plenary is smart enough to see that the file name in question is a uri and thus it just returns the uri instead of the relative path. See local function _normalize_path(filename, cwd)

local is_uri = function(filename)
  return string.match(filename, "^%w+://") ~= nil
end
-- .
-- .
-- .
local function _normalize_path(filename, cwd)
  if is_uri(filename) then
    return filename
  end

So the long short is, Plenary says "ya I can shorten your path name to a relative path", see's that the file name is a URI and nopes out. I am still exploring that more, I can solve it but I need to figure out how to retain information enough to also match whatever I provide for the file names back to their "absolute" path.

[miversen33]

Commit c0f0226 coupled with removal of extra logging (via setting log level to default (deleting vim.g.netman_log_level) produces reasonable performance for remote file accessing on Telescope-file-browser. There is still still more optimization that can be done but I still think focus on how the metadata parser works is going to be key with improving performance.

Additional items that can be considered, allowing the API to cache metadata responses and return them in a similar sort of TTL cache to eliminate duplicate requests in a short timeframe (this will help immensely with load delay after initial load)

[akinsho]

@miversen33 been checking out this plugins development, and it's looking really cool I just wanted to add another "popular" file explorer to your list, i.e. https://github.com/nvim-neo-tree/neo-tree.nvim it uses plenary under the hood, so hopefully it might be easier to integrate. It's also designed for this sort of use case if the README is to be believed, so it might be an easy ish option to integrate with as well

[miversen33]

@akinsho I have added the request to the top pinned issue :) Once I have the performance bugs worked out of my Libuv shim, I have improved performance (and likely refactored the entire shim), patched up the ssh provider to work with the "new" standard, and documented the whole thing, I will look into neo-tree. I was briefly looking through the other listed file explorers and honestly none of the others really look attainable without core developer support from the file explorer in question

• nvim-tree
• nerdtree: this will likely run into the exact same issue nvim-tree ran into in that I cannot (currently) override/redirect vim level commands that bypass lua
• chadtree: This one may be possible with some python fuckery but that would require an external plugin for it. Not to say its impossible, just not high on my list as I will need to figure out how to do some low level python shenanigans and I am not terribly interested in trying to do that. But the core of chadtree is all using the provided python pynvim module which I have no experience with currently.

Thus I was hoping I would find another file explorer I could support relatively easily. If neo-tree relies mostly on plenary (and doesn't do some of the tricks that nvim-tree does which also relies mostly on plenary), this may be attainable! Thank you for the recommendation!

[miversen33]

Looking into how Neo-tree works, it appears that supporting it will be relatively straightforward and I do not have to shim plenary for it. That is fantastic news. I have opened a new fork of it here where I will explore how to cleanly integrate it.

[cseickel]

I'm just stumbling on this now and I think integrating this into neo-tree would be a great idea. I would think that adding "netman" as a new source is probably the easiest way to do so. Feel free to reach out if you have any questions!

[miversen33]

@cseickel I looked down the rabbit hole of integrating with neo-tree and I am unsure that making a source for Netman is the best option. The goal of netman is for it to sit out of the way and act as a sort of "hidden" access point for remote resources. Because of this, idea, netman being a source means that users using neo-tree would need to explicitly say they want to access files from Netman sources as opposed to Netman seemlessly feeding neo-tree (if I understood the sources bit right when I was digging into it). That also means that users would have to maintain in their headspace what is accessed with Netman and what isn't.

My goal is to have Netman feed into neo-tree as if it were libuv (which I am working on in the issue-28-libuv-shenanigans branch right now).

Turns out, pretending to be a C interface without having access to the underlying C code, and trying to get similar performance is hard. Who knew! :upside_down_face:

[miversen33]

After alot of time digging into all of this, I have settled on the following outcome for this issue. I am working through getting a fallback put together via vim.loop shimming, though it is not going to be amazing (it will serve its bare purpose and work mostly off some buffer reading shenanigans).

This is already mostly complete on my personal branch (libruv-shenanigans) Once that is complete, I will look to get proper integration completed for Neo-tree as that is the easiest to support (via custom sources). Telescope is mostly already supported provided a resolution is reached on Issue 2071.

I will need to work through reworking alot of my testing as well as documentation (this update is an absolutely massive rework of how the API works under the hood, though the end result is mostly abiding by the predefined API spec, save for a few changes that will be called out later).

Still no time frame for completion on this however as the 80:20 rule applies heavily on this issue

[miversen33]

Just a quick update for those interested, PR #67 is floating out there right now on the libuv-branch. It is only tested with telescope-file-browser (and works ~~perfectly~~ ish with it). It does not work with neo-tree though I am currently working on writing integration for neo-tree itself instead of relying on the shim.

I don't expect anyone to look through the branch, its a hot mess lol. But I believe I have the code for this worked out, its now a matter of optimizing everything and updating my entirely broken set of tests lol.

Closing in on getting this pushed to main soon(ish ™️ )!