tantivy icon indicating copy to clipboard operation
tantivy copied to clipboard

function to fetch multiple docs at once from the docstore

Open trinity-1686a opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. In https://github.com/quickwit-oss/quickwit/pull/3226, I discovered the best access pattern to the docstore to minimize cpu usage and query duration is rather non-obvious. From inside tantivy, it's much easier to know what this pattern is, and to access things accordingly.

Describe the solution you'd like implement an async fn fetch_docs(&self, doc_ids: &[DocId]) -> crate::Result<Vec<Document>> which would figure on its own how to not decompress or read a block twice when multiple docs are inside the same doc.

[Optional] describe alternatives you've considered What was in https://github.com/quickwit-oss/quickwit/pull/3226: reorder fetches in a clever way to best leverage the existing cache. It severely lack explicitness and show of intent

trinity-1686a avatar Apr 26 '23 09:04 trinity-1686a

(unassigning myself as other people are working on it)

trinity-1686a avatar Dec 11 '23 17:12 trinity-1686a