deepset-ai
Investigate running Tools in Parallel in the ToolInvoker
Based on the feedback from @d-kleine in the discussion here it would be interesting to explore whether we could run the tools in ToolInvoker in a parallel way.
This is relevant since it's possible for an Agent to request multiple tools be called at once before continuing the generation.
Perhaps it's possible we could implement this using Threading when using Pipeline and if we add an Agent based on AsyncPipeline maybe it's possible to add a run_async method to ToolInvoker.
+1 on this, but I am a little uncertain whether LLMs would enforce an order of tool runs (inner dependencies between tools) in some scenarios, and if that's the case we have to run the tools in a sequential way (and properly handle the dependencies).
Imo, the parallel tools call would only make sense in scenarios when the data sources are independent from each other, reducing execution latency and improving resource utilization. From a business perspective, this can dramatically cut down response times in time-critical applications (while maintaining the same level of accuracy), potentially leading to better user experiences and cost savings. I have thought of use cases such as
- financial advisory system (real-time market analysis, risk assessment, portfolio analysis, etc.)
- multi-channel customer complaint resolution system (sentiment & urgency, purchase history, product return rate, etc.)
