ChatBedrock: Unable to stream with structured output

[keremnalbant]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

class Joke(TypedDict):
    """Joke to tell user."""

    setup: Annotated[str, ..., "The setup of the joke"]
    punchline: Annotated[str, ..., "The punchline of the joke"]
    rating: Annotated[Optional[int], None, "How funny the joke is, from 1 to 10"]

async def llm_stream(topic: str):
    llm = ChatBedrock(
        model=config.BEDROCK_LLM_MODEL_ID,
        model_kwargs=dict(temperature=0.05, top_k=100, top_p=0.95),
        provider="anthropic",
    )
    structured_llm = llm.with_structured_output(Joke)
    
    async for chunk in structured_llm.astream(f"Tell me a really long joke story about {topic}"):
        print(chunk)

Error Message and Stack Trace (if applicable)

No response

Description

Seems like there is something going wrong if you want to stream with structured output using ChatBedrock. Using ChatOpenAI works with the same code but ChatBedrock doesn't. Even I use astream or stream method, it just gives the output in one chunk and behaves like synchronous invocation.

The same code works for ChatOpenAI. Also tried to switch to json_schema, but doesn't help. If you remove structured_output, streaming works.

I guess for now, as a workaround, I can do some prompt engineering to get a json output.

System Info

System Information

OS: Darwin OS Version: Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:11 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6020 Python Version: 3.11.2 (main, Jul 19 2024, 17:09:07) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.3.12 langchain: 0.3.4 langsmith: 0.1.136 langchain_aws: 0.2.2 langchain_text_splitters: 0.3.0 langchain_weaviate: 0.0.3

Optional packages not installed

langgraph langserve

Other Dependencies

aiohttp: 3.10.10 async-timeout: Installed. No version info available. boto3: 1.35.36 httpx: 0.27.0 jsonpatch: 1.33 numpy: 1.26.4 orjson: 3.10.9 packaging: 24.1 pydantic: 2.9.2 PyYAML: 6.0.2 requests: 2.32.3 requests-toolbelt: 1.0.0 simsimd: 3.7.7 SQLAlchemy: 2.0.36 tenacity: 8.5.0 typing-extensions: 4.12.2 weaviate-client: 4.9.0