vim-notes is very slow with large amounts (700) of notes

[elliottw]

700 notes largest is 56kb retina macbook pro 2.4ghz core i5, 8gb ram vim 7.4.something it's weird because at around 150 notes it was blazing and now it takes several seconds to scroll a few lines.

[xolox]

Hi Elliot,

I have 1624 notes (12 megabytes in total) and it's still blazing for me :-)

You say scrolling is very slow. That doesn't really sound related to the number of notes you have. The size of the individual notes seems more relevant: Do you only see the problem in large notes or also in small notes? Did you recently upgrade vim-notes, and could the problems have started then? Also, which version of vim-notes are you running?

[elliottw]

That's why I think there is a bug! So I added about 100 files today, and I was wondering if one of them is corrupt causing this. So, it does lag a lot during scrolling of files even in the 100 line size range. It also lags in the ShowTaggedNotes page. And if I had to guess, I'd say the lag is worse when in files with a lot of special character (like how you turn bullets into middots like this: •).

On Mon, Oct 27, 2014 at 9:11 PM, Peter Odding [email protected] wrote:

Hi Elliot,

I have 1624 notes (12 megabytes in total) and it's still blazing for me :-)

You say scrolling is very slow. That doesn't really sound related to the number of notes you have. The size of the individual notes seems more relevant: Do you only see the problem in large notes or also in small notes? Did you recently upgrade vim-notes, and could the problems have started then? Also, which version of vim-notes are you running?

— Reply to this email directly or view it on GitHub https://github.com/xolox/vim-notes/issues/90#issuecomment-60696244.

[xolox]

So I added about 100 files today, and I was wondering if one of them is corrupt causing this.

I don't see how the contents of one note could cause slow scrolling in other notes (just FYI) unless you have all of those notes loaded into Vim at the same time.

So, it does lag a lot during scrolling of files even in the 100 line size range.

Okay so the size of the individual notes is probably not related then. Can you answer my other questions? In my previous message I asked:

Did you recently upgrade vim-notes, and could the problems have started then? Also, which version of vim-notes are you running?

I ask specifically because I fairly recently added 1dcfacdf16bb373dcbc730682ea0d0cbc55ecb18 and I could imagine this causing side effects that previously went unnoticed (this does assume interaction with another plug-in).

[elliottw]

Sorry, I did a clean install of vim-notes last week, I can't remember when. I've actually given up on vim-notes. Hopefully the speed issue was just with me.

On Mon, Nov 3, 2014 at 4:42 PM, Peter [email protected]:

So I added about 100 files today, and I was wondering if one of them is corrupt causing this. I don't see how the contents of one note could cause slow scrolling in other notes (just FYI) unless you have all of those notes loaded into Vim at the same time. So, it does lag a lot during scrolling of files even in the 100 line size range.

Okay so the size of the individual notes is probably not related then. Can you answer my other questions? In my previous message I asked: Did you recently upgrade vim-notes, and could the problems have started then? Also, which version of vim-notes are you running?

I ask specifically because I fairly recently added1dcfacdand I could imagine this causing side effects that previously went unnoticed (this does assume interaction with another plug-in).

— Reply to this email directly orview it on GitHub.

[JonathanReeve]

I'm having this issue, too. I have around 1100 notes. Everything (scrolling, changing to insert mode, folding, changing indent levels of bullet points) is super slow with a lot of notes. I tried changing my notes directory to an empty one, and it's blazingly fast again. When I add ~500 notes to the directory, it gets about half as slow. So I think it's related to the number of notes. Although I'm not sure how.

[JonathanReeve]

I did some profiling and came up with this:

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
    1   0.765927   0.006074  xolox#notes#edit()
   14   0.326105   0.115523  xolox#notes#highlight_names()
  646   0.300943   0.034165  xolox#notes#foldexpr()
    1   0.293516   0.078149  xolox#notes#save()
   15   0.292403   0.000629  <SNR>54_on_window_changed()
   61   0.266778             xolox#notes#inside_snippet()
   16   0.254109   0.010438  airline#update_statusline()
   16   0.243487   0.001962  <SNR>55_invoke_funcrefs()
    2   0.223508   0.002884  <SNR>16_SynSet()
   14   0.217009   0.001512  xolox#notes#refresh_syntax()
    1   0.199107   0.024713  xolox#notes#select()
   16   0.199105   0.032589  251()
    1   0.173910   0.031307  xolox#notes#get_fnames_and_titles()
   80   0.122188   0.002800  <SNR>81_get_seperator()
19412   0.109178             <SNR>22_sort_longest_to_shortest()
   14   0.106343   0.106183  xolox#notes#highlight_sources()
  364   0.103870   0.032178  airline#highlighter#get_highlight()
 2210   0.100521   0.045022  <SNR>22_words_to_pattern()
    3   0.090544   0.021763  xolox#notes#get_titles()
   96   0.085422   0.005000  <SNR>81_get_transitioned_seperator()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
   61              0.266778  xolox#notes#inside_snippet()
   14   0.326105   0.115523  xolox#notes#highlight_names()
19412              0.109178  <SNR>22_sort_longest_to_shortest()
   14   0.106343   0.106183  xolox#notes#highlight_sources()
    1   0.293516   0.078149  xolox#notes#save()
    1   0.065940   0.065416  xolox#notes#tags#scan_note()
  728              0.057498  <SNR>77_get_syn()
 2270              0.056712  xolox#misc#escape#pattern()
    2   0.052862   0.052307  xolox#notes#get_fnames()
 1104   0.068354   0.050956  xolox#notes#fname_to_title()
 2210   0.100521   0.045022  <SNR>22_words_to_pattern()
  646   0.300943   0.034165  xolox#notes#foldexpr()
   16   0.199105   0.032589  251()
  364   0.103870   0.032178  airline#highlighter#get_highlight()
    1   0.173910   0.031307  xolox#notes#get_fnames_and_titles()
   19              0.025575  <SNR>6_rtp_rm_a()
    1   0.199107   0.024713  xolox#notes#select()
    4   0.023621   0.022622  xolox#shell#highlight_urls()
    3   0.090544   0.021763  xolox#notes#get_titles()
  112              0.020967  <SNR>81_get_accented_line()

Hope that helps.

[JonathanReeve]

Could it have something to do with this?

FUNCTION  <SNR>22_sort_longest_to_shortest()
Called 9706 times
Total time:   0.039217
 Self time:   0.039217

count  total (s)   self (s)
                              " Sort note titles by length, starting with the shortest.
 9706              0.031467   return len(a:a) < len(a:b) ? 1 : -1

In the ~5 second test I did that generated the above profile, this function was called 9706 times. Is there a better / faster sort algorithm that would speed this up?

[JonathanReeve]

Sorry for the torrent of comments, but I'm trying to track down the source of this issue. I think it might be something related to a long note, as well, because I seem to get this slowdown when: a) I have a lot of notes, and b) I'm editing a long note, say, with lots of sections and lots of bullet points.

[pdoak]

I am experiencing a similar slow down when a note gets to over about 40 lines. Did you ever manage to solve the problem?

[peplin]

Could this be related to #82? I just tried updating to the latest commit in vim-notes, and it still has the same issue - very high CPU usage when moving the cursor around. I have 372 notes. I had narrowed down the issue to starting with 677a27fecc4aef6acbbb6b3ed68d6066a5daa7c3.