Anthropic chat completion missing `assistant` role

[flipace]

trafficstars

What Happened?

When using anthropic with langchain, langchain won't be able to parse out tool calls since it depends on the role of the response delta to determine whether there actually can be such a prop.

I noticed that in https://github.com/Portkey-AI/gateway/blob/cc328ee2260f990a39e25d1eba675235daddf2d0/src/providers/anthropic/chatComplete.ts the returned delta is missing the assistant role.

I'm not 100% sure whether that's the root cause of the issue, but fairly certain. The place where langchain js checks this role can be found here: https://github.com/langchain-ai/langchainjs/blob/e2de559a4c4fea9ef4c633e8afaef774f2263299/libs/langchain-openai/src/chat_models.ts#L1828 (since we should use the langchain openai class as per portkey docs)

With portkey, it ends up in the fallback case, returning a message without tool calls.

What Should Have Happened?

Portkey should assign the proper role to the returned chunks for anthropic, so that langchain can properly handle it.

Relevant Code Snippet

No response

Your Twitter/LinkedIn

No response

[narengogi]

thanks for the report @flipace

though the OpenAI API spec does not have role in streaming chunks, it is being sent in the first chunk in OpenAI streaming responses

ideally portkey gateway should handle this.

initial thoughts:

• Sending the role key only in the first chunk might be a problem because the stream transformers are largely stateless and are not aware of the chunk number.
• Sending the role key in all the chunks would be unfaithful to the OpenAI response structure but is a feasible option. (I'm worried if any framework has strict type checks on the response object structure, it would be a problem)