haystack
haystack copied to clipboard
Pipeline eval should use batch processing for queries
We recently added a run_batch() method in every pipeline node, which can increase inference speed when batching multiple queries: https://github.com/deepset-ai/haystack/pull/2481
However, pipeline.eval() internally still calls pipeline.run() and evaluation is slow. We should change that so that pipeline.eval() calls pipeline.run_batch() and thereby increase evaluation speed.
@Timoeller brought up this issue
This would be greatly helpful to me as well