nvim-lua
grep_string or live_grep etc can lead to hang and excessive memory use for noobs.
I'm on Windows and I've installed Neovim and added Kickstart.nvim, rg, fd and the steps for telescope-fzf-native.
Life is good and It feels like there's nothing at all between me and my text documents. I even dreamt I was doing vim motions.
I'm playing around with all the new stuff with init.lua open and I do <leader>sw which is grep_string aka '[S]earch current [W]ord'. My cursor was on a { character so it starts churning through my whole home folder (1TB~ of files) looking for {. nvim.exe quickly starts becoming unresponsive as it's memory usage quickly grows to over 50gb at 97% of the system's memory usage. Watching in task manager eventually the disk usage on nvim.exe start to climb up as gigabytes of memory pages are written to disk. The same is true for <leader>sg aka '[S]earch by [G]rep' and typing. Within seconds of doing this nvim becomes unresponsive and needs to be forced closed.
I'm not too sure how to introspect this behavior to see what exactly is the cause. Whether its the number of results, a very large file, binaries or w.h.y. I'm not sure. But I imagine its not unexpected that this could happen given the parameters like greping 1tb of files with no file type/file size like filters.
Would safeguarding against this kind of behavior by having default file size limits or file type filters for these searches make sense to be included in Kickstart.nvim somehow?
You went looking for every instance of a specific ASCII character across over 1,099,511,627,776 bytes of content (quick stats [1]: that's likely over 7,000,000 files). That list of matches is contextualised with the line they were found at. You browsed that list, requesting for each entry, the entire context to be displayed and highlighted in a panel on the side. The memory consumption went way up. The match results counter also was continuously going up.
I can reproduce this!
Would safeguarding against this kind of behavior by having default file size limits or file type filters for these searches make sense [...] ?
To best answer the first part of your question, I think a bit more context is necessary:
- how long had you been playing with that list before your system ran out of memory? Are we talking milliseconds/very few seconds, or actually recreational behaviour that went on for over a minute, with the indicators (like the matches counter) going up, up, up? On my end, it takes well over a minute.
- was the memory freed for your system soon after you aborted your search? On my end, it was cleared virtually instantly.
Would safeguarding against this [...] make sense [...] to be included in Kickstart.nvim somehow?
I'd say... no. I think this project is about introducing people to Neovim, and perhaps to some extent Vim; it's not really about offering a complete, opinionated book on how to best live your text-editing life. And even if it were, this specific example here seems to be quite an arcane opinion about a quite esoteric topic!
[1] The "quick stats" from the largest human-managed home directory I had ready access to:
; fd 2>/dev/null --type f | wc -l
1676789
; fd 2>/dev/null --type f -0 | du --files0-from=- -ch | tail -1
243G total
; bc <<< 'scale=1;1024/243*1676781'
7042480.2
My point was since Kickstart is somewhat an introduction, hitting full memory usage leading to low disk space might be a poor intro especially to someone who might not be aware of system resource usage. When disk space gets low on Windows machines, chromium based apps start deleting temp data (you get signed out of everything, cookies etc) among other weird behavior like sqlite corruption if you have anything using that at the same time. If someone already has low disk space it could be problematic if not dealt with proactively. Sane defaults are opinions I suppose and that is partly what I was looking for in Kickstart. The scenario I laid out shows how I easily fumbled into it and I was surprised how fast it escalated. After trying 10~ times I can reproduce this about 1/3rd of the time.
- how long had you been playing with that list before your system ran out of memory?
- Out of memory in less than 10 seconds but unresponsive sometimes instantly. Tapping esc doesn't get me out. It can take about 30 seconds before it starts writing to disk but on my ssd there's no trouble in writing many gigabytes per second. I've always stopped it after it's used 50~gb on disk.
- was the memory freed for your system soon after you aborted your search?
- In most cases yes, but not always. It often frees on the next search. The issue is not the memory consumption but it hanging then potentially filling the disk.
I think I brought some personal anger with me this morning when I answered this thread: my apologies, @1337cookie, I could have been far nicer. And I really didn't think just letting it look for searches for 10 seconds would fill 50GB of RAM, there's definitely something I don't understand there. If you want to reopen it to let others share their thoughts, it's fine by me: I'm not all that in sync with what the scope of this project is: we all eventually graduate away from it :) Cheers!
@ccjmne Your comment was fair and I read it as an attempt to rule out if I'm just a complainer or if there was some merit to what I'm saying. I didn't give a lot of context on the timings of things which would've been useful. I have a very fast 7800X3D cpu which has a 96mb L3 cache and 7000~mb/s NVMe SSD so I think my machine is well suited to eating memory when it needs to like above.
So I was checking out the possible technical reasons, and solutions and I found:
- grep_string and live_grep explicitly require "ripgrep" downloaded (unless there is a fall back option to plain old "grep", because I haven't checked it then pointing this out is a mistake on my part).
- "ripgrep" is a rust based implementation and supposed improvement of "grep" which is written in native C, and rust based cli tools are generally a little less efficient than C based cli tools.
- The problem is...and I feel this is a very ironic and bad thing to say..."telescope.nvim"
- There have complains all across the neovim community that telescope is too slow especially on a very large codebase, and that can be expected to be transferred specifically to grep_string and live_grep at a greater magnitude.
- I know kickstart.nvim and telescope.nvim are both creations of TJDeVries so I feel bad at suggesting the next point.
- Move to fzf-lua.nvim, which has a fallback option to plain old simple "grep", and is generally faster and is also adopted officialy now by LazyVim.
So if possible switch to fzf-lua.nvim, though this will need major changes across the codebase.
Distributions like LazyVim have also adopted fzf-lua and thePrimeagen is also considering moving away from telescope.
