P2-Batch Inference

[ronakice]

Currently, we only support inference with a (single query, single subset of documents), but technically we could batch over the query dimension pretty easily, doing it over document subsets is not doable given that the next call is not inferable in the sequential process that is sliding windows.