paper-qa icon indicating copy to clipboard operation
paper-qa copied to clipboard

Summarisation should use batch methods

Open zacps opened this issue 2 years ago • 0 comments

Currently summarisation makes one call to the underlying LLM per document to summarise.

Instead aget_evidence should use batch methods (agenerate/aapply) to produce a single LLM call.

This could provide significant speedups for local LLMs.

zacps avatar Aug 07 '23 00:08 zacps