rai icon indicating copy to clipboard operation
rai copied to clipboard

feat: Dual agent + tracing tags

Open jmatejcz opened this issue 7 months ago • 0 comments

Purpose

Enable usage of agents that compose of 2 llms. This can be usefull for testing the model which does not have tool calling capabilities. In such case we can use it in combination with llm that can make these calls.

Also this PR proposes changes to tracing to help debug and check results

Proposed Changes

  • New way of creating agents -> create_multimodal_to_tool_agent function. It defines 2 llm like stated above, with addition of context bridge that removes images from context when passing it to 2nd model. This approach is needed when testing the 1st model on spatial reasoning tasks, as we ensure that the tool call is based only on 1st model's answer.

I know it's not a general solution for pairing llms but i think every task would require slightly different approach to defining the langchain workflow. So when new workflow will be needed, just build something similar like in proposed function.

  • new example file for showing how to run dual agent

  • added tags to tracing for both tool-calling-agent and manipulation-o3de benchmarks

  • recursion limit in tool-calling-agent now scaling with number of langchain nodes, as more nodes = more recursion

Issues

  • Links to relevant issues

Testing

  1. Run with tracing enabled
python src/rai_bench/rai_bench/examples/dual_agent.py 
  1. check the traces, if they are tagged clearly

jmatejcz avatar May 29 '25 11:05 jmatejcz