langchain stream: true and tools infinite loop

I'm exploring Langchain 0.4.0 on a new pet project and I plugged in Scaleway Generative API as an OpenAI compatible endpoint.

I'm using llama 3.3 70b model and I think I'm facing a bug

When I use stream: true without tools, everything works as expected and I get new messages streamed into the console When I use tools with stream: false, the tools are being called and used in the LLM's response

BUT when I want to combine the usage of tools with stream: true, I'm getting an infinite loop on the tool's call.

I'm calling my run with LLMChain.run(mode: :while_needs_response) I added the an IO.inspect on the on_llm_new_delta callback to see what's happening and I'm seeing an infinite loop over the following:

%LangChain.MessageDelta{
  content: "",
  merged_content: [],
  status: :incomplete,
  index: 0,
  role: :assistant,
  tool_calls: nil,
  metadata: nil
}
%LangChain.MessageDelta{
  content: nil,
  merged_content: [],
  status: :incomplete,
  index: 0,
  role: :unknown,
  tool_calls: [
    %LangChain.Message.ToolCall{
      status: :incomplete,
      type: :function,
      call_id: "chatcmpl-tool-5c2821496b704465853987b94d23370c",
      name: "sections_features",
      arguments: nil,
      index: 0
    }
  ],
  metadata: nil
}

As a comparison, here is what gets logged on a working stream:

...
%LangChain.MessageDelta{
  content: " include",
  merged_content: [],
  status: :incomplete,
  index: 0,
  role: :unknown,
  tool_calls: nil,
  metadata: nil
}
%LangChain.MessageDelta{
  content: "?",
  merged_content: [],
  status: :incomplete,
  index: 0,
  role: :unknown,
  tool_calls: nil,
  metadata: nil
}
%LangChain.MessageDelta{
  content: "",
  merged_content: [],
  status: :complete,
  index: 0,
  role: :unknown,
  tool_calls: nil,
  metadata: nil
}

Is there anything I'm doing wrong ? is this a bug ? (happy to dive into the code and submit a PR if that's the case and if you could point me out to what should be fixed.

Jun 14 '25 18:06 jfayad

Hi @jfayad!

That sounds like a bug or incompatibility with our OpenAI module and the Scaleway API.

I don't have any experience with that service.

For debugging something like this, here's the approach I take:

setup a minimal text that makes a live API call
enable verbose_api: true on the ChatOpenAI module
see what's return from the actual API call and begin diagnosing where the mismatch is.

Oct 01 '25 00:10 brainlid

Thank you for the tip.

I enabled verbose api and could see the LLM is replying with empty content.

I removed the mode: in the run call to avoid a loop and here is the chunk I'm receiving:

RCVD RAW CHUNK: "data: {\"id\":\"chatcmpl-d8fab86a-564d-4dc6-9396-dd36beb967b3\",\"object\":\"chat.completion.chunk\",\"created\":1761907335,\"model\":\"llama-3.3-70b-instruct\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"username\\\": \\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-d8fab86a-564d-4dc6-9396-dd36beb967b3\",\"object\":\"chat.completion.chunk\",\"created\":1761907335,\"model\":\"llama-3.3-70b-instruct\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"user\"}}]},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-d8fab86a-564d-4dc6-9396-dd36beb967b3\",\"object\":\"chat.completion.chunk\",\"created\":1761907335,\"model\":\"llama-3.3-70b-instruct\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"123\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-d8fab86a-564d-4dc6-9396-dd36beb967b3\",\"object\":\"chat.completion.chunk\",\"created\":1761907335,\"model\":\"llama-3.3-70b-instruct\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\"}}]},\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"stop_reason\":128008}]}\n\ndata: [DONE]\n\n"

If I then re-add the mode: :while_needs_response

I can see the tool being called with the correct argument and get to the end of it but then it fails to stop:

RAW DATA BEING SUBMITTED: %{
  messages: [
    %{
      "content" => [
        %{
          "text" => "You are an ID generator, at first greet.\nOnce asked for an ID use the random_id tool to return a proper ID as a message. if you are missing information ask it from the user.\nIf for any reason you are not able to use the tool, provide the list of tools you have detected in your response\n\nOutput format:\n- ALWAYS respond with a single JSON object.\n- One line only. No code fences, no extra text.\n- Shape:\n{\"widget_type\":\"<one of ALLOWED>\",\"message\":\"<your message>\",\"last_step\":<false>}\n\nConstraints:\n- `\"widget_type\"` MUST be one of: [\"userInput\",\"userUpload\"]\n- `\"message\"` MUST be written in en\n- `\"last_step\"` MUST be a boolean false.\n",
          "type" => "text"
        }
      ],
      "role" => :system
    },
    %{
      "content" => [
        %{"text" => "Give me a new ID for user jooj", "type" => "text"}
      ],
      "role" => :user
    },
    %{
      "content" => [],
      "role" => :assistant,
      "tool_calls" => [
        %{
          "function" => %{
            "arguments" => "{\"username\":\"jooj\"}",
            "name" => "random_id"
          },
          "id" => "chatcmpl-tool-8ff838af123f4a118243251985e99400",
          "type" => "function"
        }
      ]
    },
    %{
      "content" => [
        %{"text" => "{\"id\":\"jooj-1761907567692926000\"}", "type" => "text"}
      ],
      "role" => :tool,
      "tool_call_id" => "chatcmpl-tool-8ff838af123f4a118243251985e99400"
    }
  ],
  stream: true,
  tools: [
    %{
      "function" => %{
        "description" => "Based on a provided username return a random custom id that respects internal criterias.",
        "name" => "random_id",
        "parameters" => %{
          type: "object",
          required: ["username"],
          properties: %{
            username: %{
              type: "string",
              description: "The username to generate an ID for. It should be provided by the user."
            }
          }
        }
      },
      "type" => "function"
    }
  ],
  n: 1,
  model: "llama-3.3-70b-instruct",
  max_tokens: 2000,
  temperature: 1.0
}
RCVD RAW CHUNK: "data: {\"id\":\"chatcmpl-9605006e-9a07-4eaa-adef-f72f93da755f\",\"object\":\"chat.completion.chunk\",\"created\":1761907574,\"model\":\"llama-3.3-70b-instruct\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}]}\n\n"
READY TO PROCESS: [
  %{
    "choices" => [
      %{
        "delta" => %{"content" => "", "role" => "assistant"},
        "finish_reason" => nil,
        "index" => 0,
        "logprobs" => nil
      }
    ],
    "created" => 1761907574,
    "id" => "chatcmpl-9605006e-9a07-4eaa-adef-f72f93da755f",
    "model" => "llama-3.3-70b-instruct",
    "object" => "chat.completion.chunk"
  }
]

I don't mind getting my hands "dirty" to fix this but I'm not quite sure where should I start looking at to debug this. Is it the finish_reason or the delta->content that is wrong here ? And why it isn't able to conclude ? I mean we've received almost everything from the LLM calls (extracting the arg to the tool call, calling the tool and providing the data back to the LLM in the request, so why aren't we receiving a correct answer ?)

Oct 31 '25 10:10 jfayad