Update documentation on how to make a streaming callback
Original request in this comment
I think it would be great if we could update our streaming docs here to show our current
print_streaming_chunkimplementation instead of the generic one we currently have.
I think specifically explaining to users how to stream different different aspects of LLM generation and Tool calling. Right now we have three distinct types of StreamingChunks which are normal text generation (last if statement), Tool Call(s) (first if statement) and Tool Result (second if statement).
Then I think it'd make sense to update each of our ChatGenerator docs (e.g. OpenAIChatGenerator) to link to the above section (Streaming Support)
Here is the new version of print_streaming_chunk
def print_streaming_chunk(chunk: StreamingChunk) -> None:
"""
Callback function to handle and display streaming output chunks.
This function processes a `StreamingChunk` object by:
- Printing tool call metadata (if any), including function names and arguments, as they arrive.
- Printing tool call results when available.
- Printing the main content (e.g., text tokens) of the chunk as it is received.
The function outputs data directly to stdout and flushes output buffers to ensure immediate display during
streaming.
:param chunk: A chunk of streaming data containing content and optional metadata, such as tool calls and
tool results.
"""
# Print tool call metadata if available (from ChatGenerator)
if chunk.meta.get("tool_calls"):
for tool_call in chunk.meta["tool_calls"]:
if isinstance(tool_call, ChoiceDeltaToolCall) and tool_call.function:
if tool_call.function.name and not tool_call.function.arguments:
print(f"[TOOL CALL - {tool_call.function.name}] ", flush=True, end="")
if tool_call.function.arguments:
if tool_call.function.arguments.startswith("{"):
print("\nArguments: ", flush=True, end="")
print(tool_call.function.arguments, flush=True, end="")
if tool_call.function.arguments.endswith("}"):
print("\n\n", flush=True, end="")
# Print tool call results if available (from ToolInvoker)
if chunk.meta.get("tool_result"):
print(f"[TOOL RESULT]\n{chunk.meta['tool_result']}\n\n", flush=True, end="")
# Print the main content of the chunk (from ChatGenerator)
if chunk.content:
print(chunk.content, flush=True, end="")
@dfokina This is still blocked by changes we plan to make as part of https://github.com/deepset-ai/haystack/issues/9347 Better wait with working on it.
Partially done, will wait for the other issue to be closed to finish up the remaining Chat Generators