Handle paging

[phillro]

Will try paging using MR or inverse posting doc.

[mroushdy]

try this

ex.search({name:'sample'},{mongo:query}||false, {}, {score:true,naturalstop:true, skip: 20, limit: 20 }, function (err, res) { if (err) { console.log(err); } else { console.log(JSON.stringify(res)); }
})

[phillro]

Sorry, let me clarify.

The issue is to handle sort/paging with scores enabled. As scoring is done after the results are returned, sorting/paging will be very inefficient with large numbers of documents.

I plan on trying to support this by trying an inverse posting document.

[mroushdy]

oh yes I just looked at the code, you are right. I am a bit new to node but if you point me in the right direction I could try to have a go at this.

[phillro]

Scoring is currently handled in lib/search.js, method addScores.

This plan is still a little half baked, there are two ways I was thinking of doing this and both would require some experimentation.

First idea: Create a second collection that would either be updated periodically or at document save time. This collection would contain inverse term frequencies & scores. So each set of N-grams would have a posting document containing a list of all document ids and the n-gram set's frequency and scores. Search would then wrap two queries, (1) Getting the document ids sorted by score, (2) Retrieving the document.

The second idea: Using a map reduce function, generate the previously mentioned inverse term freq results, returning the list of document ids & scores. Either store output the MR to a temp collection or them and cache the ids and scores in the client to provide paging. I think that this would be more elegant, but I'm not sure mongo would let it be done right.