feat: Dual agent + tracing tags
Purpose
Enable usage of agents that compose of 2 llms. This can be usefull for testing the model which does not have tool calling capabilities. In such case we can use it in combination with llm that can make these calls.
Also this PR proposes changes to tracing to help debug and check results
Proposed Changes
- New way of creating agents ->
create_multimodal_to_tool_agentfunction. It defines 2 llm like stated above, with addition of context bridge that removes images from context when passing it to 2nd model. This approach is needed when testing the 1st model on spatial reasoning tasks, as we ensure that the tool call is based only on 1st model's answer.
I know it's not a general solution for pairing llms but i think every task would require slightly different approach to defining the langchain workflow. So when new workflow will be needed, just build something similar like in proposed function.
-
new example file for showing how to run dual agent
-
added tags to tracing for both tool-calling-agent and manipulation-o3de benchmarks
-
recursion limit in tool-calling-agent now scaling with number of langchain nodes, as more nodes = more recursion
Issues
- Links to relevant issues
Testing
- Run with tracing enabled
python src/rai_bench/rai_bench/examples/dual_agent.py
- check the traces, if they are tagged clearly