vimflowy
vimflowy copied to clipboard
WuTheFWasThat
Reverseindex search
Makes search fast for large documents.
The main constraint is database access time with firebase. On average it takes 3 database queries to maintain the reverse index every time a row is changed, which is pretty much the limit before typing feels sluggish.
To get the number of queries down, there are compromises. You can only search for whole words, you must type out the entire word. Only 20,000 rows are stored for any word, so the search will fail in edge cases where every search query is super common.
todos: show initialization progress rewrite search tests
yeah I agree... do you think updating the datastore only after the user stops typing would work?
that's a good idea, can even just only update after they leave a row, maybe? and you can also debounce the rest of the updates
(it's hard to guarantee the index is 100% correct, right? i'd be okay with small race conditions that get fixed when you revisit the row)
I changed it to update after leaving the row. None of the updates are blocking, so there might be some race conditions leading to an inconsistent index. The search function is robust to false positives since it manually checks it again. All new rows in the last session are updated again in the next session so there shouldnt be false negatives.
Also since timing isnt as critical now it indexes all prefixes of words. The main performance bottleneck now is getting the search results text for display which seems unavoidable? Maybe debounce the search function itself or only search on enter?
I'm having a lot of trouble debouncing search. Do you see how it could be done? Also still have no idea why rendering results takes so long :/ Would appreciate any help if you have time!
i think you want to debounce this line in menu.ts
this.results = await this.searchFn(query);
it looks like it might be straightforward, but i wouldn't be surprised if it ended up very tricky
regarding the rendering results, you can profile in chrome i think. is it worse on your branch than on master?
I think this pr is ready to be reviewed, I'll leave debounce for the future. In its current state this pr makes search very usable for large documents, just typing search queries is a bit sluggish.
When testing I recommend making a backup of your data, it will add a lot of stuff to your database. It also takes a while to fully initialize, minutes for me.