need to wait until finder command is done

[antoinemadec]

Description

When a finder takes multiple seconds to generate all the entries, there are 2 issues:

1. the screen refresh is frozen unless we move the cursor (up/down arrow)
2. the sorter does not take effect until the command is (almost) done

Neovim version

NVIM v0.8.0-dev
Build type: Release
LuaJIT 2.1.0-beta3

Operating system and version

ManjaroLinux 21.2.6

checkhealth telescope

telescope: require("telescope.health").check()
========================================================================
## Checking for required plugins
  - OK: plenary installed.
  - WARNING: nvim-treesitter not found.

## Checking external dependencies
  - OK: rg: found ripgrep 13.0.0
  - OK: fd: found fd 8.3.2

Steps to reproduce

1. cd <big_project> (e.g.: neovim git repo)
2. nvim --clean -u minimal.vim
3. :lua require("telescope.builtin").grep_string({search = ""})
4. type pattern in prompt, use up and down arrow

I tried with and without telescope-fzf-native.nvim and observed the same behavior

Expected behavior

I expect the screen to be refreshed without user input and the sort to take effect immediately. This is how fzf.vim deals with commands that take time.

It should behave like (even if slower than) this command: rg --color=never --no-heading --with-filename --line-number --column --smart-case -- "" | fzf

Actual behavior

1. the screen refresh is frozen unless we move the cursor (up/down arrow)
2. the sorter does not take effect until the command is (almost) done

Minimal config

set rtp+=/home/antoine/.local/share/nvim/site/pack/packer/start/plenary.nvim
set rtp+=/home/antoine/.local/share/nvim/site/pack/packer/start/telescope.nvim
lua << EOF
require('telescope').setup()
EOF

[fdschmidt93]

For grep_string telescope does essentially everything (processing rg output, sorting, drawing, ...) on the neovim main thread in lua. There was an essentially similar note on reddit recently about telescope performance in large find_files (50K+) that essentially reports the same thing, see my reply to this here which goes into bit more context.

tl;dr: telescope does everything single threaded in luajit vis-a-vis fzf.vim, which essentially is an external process doing everything in highly optimized go multi-threaded.

the sorter does not take effect until the command is (almost) done

I'd have to take a look at the loop again, but I'm pretty sure that's mostly because the main thread is clogged with tasks and freezes. Appropriate parallelization would be the most important lever here.

Intermittently, #1491 will yield the largest improvements, which would make grep_string just as usable as fzf for something between 100-500K entries (depends on your query). I've just tried it on neovim repo and results are unfortunately quite sobering.

If you don't need fuzzy matching but just "and" operator this input cb practically gives me speed parity for what I'm actually using (intersecting results by query terms). If you really need fzf as such (large-scale fzf-style searches over 500K to open ended number of lines), I suppose fzf.lua would be your best bet.

[antoinemadec]

@fdschmidt93 thanks for the quick response.

Do you mean that even when fps_mode will be implemented, the result will still be slow on big repos ? Or did you mean that searching in big repos is slow today ?

I work in HW design/verification, so the need to parse huge files and tags is unfortunately not a weird flex on my end.

I actually wanted to re-implement an efficient tags picker : my tag file is currently ~1GB and takes 1min to show in telescope. After some optimization in the code (Path:new(vim.fn.expand(ctags_file, true)):iter() is really killing the performances), I still have to wait 15sec for telescope tags to show up.

So I thought of using cat ./tags instead but found out about the problems I am listing in this ticket.

I guess I will add a live_tags in my extension : no fuzzy find, but it is very snappy.

[Conni2461]

pro tip. If telescope doesn't work for you just use fzf.vim. Its a great tool

Your issue is a duplicate of https://github.com/nvim-telescope/telescope.nvim/issues/1423

[antoinemadec]

@Conni2461 , I'm sorry if I upset you, that was not my intent. fzf.vim is actually where I was coming from, I even wrote a plugin for it: https://github.com/antoinemadec/coc-fzf

I got seduced by 2 things in telescope.nvim:

• the looks
• the cleaner / less messy / more compartmentalized code

Will fps_mode gonna fix the issue described in this ticket ?

Btw, here is my implementation of live_tags: https://github.com/antoinemadec/telescope-git-browse.nvim/blob/0858d05f96c3f9091d1d0dda91a8713a91a119fc/lua/telescope/_extensions/git_browse.lua#L108-L157

[fdschmidt93]

I even wrote a plugin for it: https://github.com/antoinemadec/coc-fzf

At least I'm aware -- kudos, I loved that coc extension back when I was using coc :)

Will fps_mode gonna fix the issue described in this ticket ?

It depends. I'll be honest most likely not what you are expected from fzf seamlessly fuzzily searching 1M+ lines. In telescope, you can do that via rg-repping.

fps_mode will enable that the entry_manager (that sorts scored entries) more lazily renders items in order in the results buffer. I would have to check the code in particular, I'm not sure whether that leads to optimizations per se already (maybe b/c of lower tick rate).

The second thing is rendering is more lazy (think "60 fps") and can be tuned.

Both in combination open avenues for lots of speed improvements in a single thread alone. I'd expect 100K entries to be sorted fluently then in fzf-style search (recalling from tests).

To truly match fzf, telescope will have to have parallelization. Maybe eventually we'll have rust-based backend or something, but that's a different story with its own complexities.

Long story short, if you truly have these performance requirements (again, on fzf-style search), then again, my honest recommendation would be fzf.lua over telescope (though see my Reddit reply why at least I quite like telescope better :).

[antoinemadec]

Would a live_tags PR be of interest on this project ?

For info, I have been using it at work on ~1GB tag files for the past couple of days. live_grep-ing tags ended up being more efficient for my workflow than grepping the whole tag file in fzf.vim.

[Conni2461]

Nope i dont think a live_tags is currently interesting for this Project. Also see https://github.com/nvim-telescope/telescope.nvim/issues/1228

But i am interested in merging the cat improvement for the current tags picker. Are you interested in opening a PR?

[Conni2461]

I flagged this as performance issue. This will be a "long term issue". fps will not fix all issues but it will help us make other changes in the future that might mitigate or even fix this issue.

So this will not be fixed for 0.1 and will probably require a native component.