Agent API Streaming - if a tool gets called twice `tool_input` and `observation` are returned for just one call.

[tognee]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Create an Agent with ddg image search. Ask the Agent to see a picture of a dog and a picture of a cat. See the result.

✔️ Expected Behavior

ddgo_img is called two times, see the query "dog" and the query "cat", see each call output.

❌ Actual Behavior

ddgo_img is called two times, you can only see the "cat" query and output.

[tognee]

The issue is that observation and tool_input uses the tool name as the key, it should use the tool_call_id instead. I think the agent_thinking event should be rethinked.

[crazywoola]

What is your model provider and it's version?

[dosubot[bot]]

Hey @tognee! I'm a bot designed to help you with bug fixes, answer questions, and assist you in becoming a contributor. I'm here to help while you wait for a human maintainer. Let's tackle this together!

To address the issue where tool_input and observation are only returned for one call when a tool is called multiple times, you can modify the implementation to use tool_call_id instead of the tool name as the key. This change will ensure that each tool call is uniquely identified, allowing for the correct association of inputs and observations for each call.

Here's how you can make this change:

1. In the save_agent_thought method, update the dictionary comprehensions to use tool_call_id as the key:

self.save_agent_thought(
    agent_thought=agent_thought,
    tool_name="",
    tool_input="",
    thought="",
    tool_invoke_meta={
        tool_response["tool_call_id"]: tool_response["meta"] for tool_response in tool_responses
    },
    observation={
        tool_response["tool_call_id"]: tool_response["tool_response"]
        for tool_response in tool_responses
    },
    answer="",
    messages_ids=message_file_ids,
)

This change will ensure that each tool call is tracked separately, resolving the issue of overlapping tool_input and observation entries for multiple calls with the same tool name ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[tognee]

What is your model provider and it's version?

tried with openai, azure openai and vertexai

but I think this is not linked with a specific provider

[dosubot[bot]]

Hi, @tognee. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

• You reported a bug in the Agent API Streaming where calling a tool twice only returns the tool_input and observation for one call.
• The issue was observed using the ddg image search tool for two different queries.
• You suggested using tool_call_id instead of the tool name as the key to resolve this.
• The issue is not linked to a specific model provider, as tested with openai, azure openai, and vertexai.
• I provided a potential solution by modifying the implementation to use tool_call_id.

Next Steps:

• Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here.
• If no further updates are provided, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

[tognee]

The issue is still present because the code hasn't been changed.

The problem is in the file fc_agent_runner.py:

• On line 129 and line 156 each tool call is appended in an array with indexes 0 for tool_call_id, 1 for tool_name and 2 for tool_arguments
• On line 130 and line 157 it's using tool_call[1], so when calling a tool like ddg_img twice it is "ddg_img;ddg_img"
• On line 133, line 137, line 160 and line 164 tool_call_inputs gets set by iterating tool_calls setting key tool_call[1] to the value tool_call[2], so tool_name to tool_arguments. Being that the name is not unique, the second iteration overwrites the first one.

Here the tool calls are being responded correctly to the agent using the actual tool_call_id as the key. But then here it uses the overwritten tool_call_inputs when saving it.

Finally here again it uses the tool_name as the key to save the tool response / observation and meta tags.

Until this hasn't been changed the issue will be present.