Adrien Grand
Adrien Grand
I found this recent paper by well-known people in the IR efficiency space quite interesting: https://arxiv.org/pdf/2405.01117. It builds on inverted indexes and simple/intuitive ideas: - BP reordering, that Ben alluded...
I'm unclear of your goal here. Since these queries only have negated clauses, they do not match anything. Are you focusing on the rewrite() cost?
> @jpountz Do you mean to update the queries to something like (-field:some *:*) (-field:text *:*) (non optimized) to (-(+field:some +field:text) *:*) (optimized) in the tasks ? I was more...
> I'm still not happy with the 2-3 K QPS :) Something seems amiss I suspect it's because the only scoring query produces a constant score. So in practice, Lucene...
Could you use tasks where dynamic pruning doesn't apply instead of disabling it? E.g. use counting tasks?
Indeed `IndexSearcher#count` has some optimizations to bypass postings. But it was mostly an example, some cheap faceting should work too?
Thanks for looking into this. I agree that it may not be an issue in practice, but I still like that we enforce thread confinment for merging. It seems that...
It does not address the `SortingCodecReader` problem, but here's a commit that does what I had in mind FWIW: https://github.com/apache/lucene/compare/main...jpountz:lucene:test/improve-cms-concurrency-testing?expand=1.
It feels too big for 9.11, maybe it's better to only keep knn vector merging parallelism in 9.11 and have this change only in 9.12?
@benwtrent Shall we close this issue as "won't fix"?