haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Pipeline eval should use batch processing for queries

Open julian-risch opened this issue 3 years ago • 1 comments

We recently added a run_batch() method in every pipeline node, which can increase inference speed when batching multiple queries: https://github.com/deepset-ai/haystack/pull/2481 However, pipeline.eval() internally still calls pipeline.run() and evaluation is slow. We should change that so that pipeline.eval() calls pipeline.run_batch() and thereby increase evaluation speed.

@Timoeller brought up this issue

julian-risch avatar Jun 07 '22 09:06 julian-risch

This would be greatly helpful to me as well

sjrl avatar Jul 19 '22 13:07 sjrl