tantivy
tantivy copied to clipboard
function to fetch multiple docs at once from the docstore
Is your feature request related to a problem? Please describe. In https://github.com/quickwit-oss/quickwit/pull/3226, I discovered the best access pattern to the docstore to minimize cpu usage and query duration is rather non-obvious. From inside tantivy, it's much easier to know what this pattern is, and to access things accordingly.
Describe the solution you'd like
implement an async fn fetch_docs(&self, doc_ids: &[DocId]) -> crate::Result<Vec<Document>> which would figure on its own how to not decompress or read a block twice when multiple docs are inside the same doc.
[Optional] describe alternatives you've considered What was in https://github.com/quickwit-oss/quickwit/pull/3226: reorder fetches in a clever way to best leverage the existing cache. It severely lack explicitness and show of intent
(unassigning myself as other people are working on it)