Fix Assistant OpenAI adapter to handle message content structure returned by to_hash method
While using the gem for the first time I was discovering Langchain::Assistant and experimenting with it.
While using naively I came across an issue :
@assistant = Langchain::Assistant.new(
llm: Langchain::LLM::OpenAI.new(
api_key: ENV['OPENAI_API_KEY'],
default_options: {
temperature: 0.7,
chat_model: 'gpt-4o-mini'
}
),
instructions: "FOO BAR ZOO",
)
@assistant.add_message(content: "Foo bar Zoo")
messages_hash = @assistant.messages.map(&:to_hash)
Here message_hash equal
[{role: "system", content: [{type: "text", text: "FOO BAR ZOO"}]}, {role: "user", content: [{type: "text", text: "Foo bar Zoo"}]}]
From there I was thinking great I can just persist messages_hash somewhere and then use the method @assistant.add_messages to resume the conversation
But this happened :
@assistant.clear_messages!
@assistant.add_messages(messages: messages_hash)
resumed_hash = @assistant.messages.map(&:to_hash)
resumed_hash equal this
[{role: "system", content: [{type: "text", text: "[{type: \"text\", text: \"FOO BAR ZOO\"}]"}]},
{role: "user", content: [{type: "text", text: "[{type: \"text\", text: \"Foo bar Zoo\"}]"}]}]
The message_hash get stringified and nested into a new hash
After some investigation I found out that this behaviour comes from the function build_message at /lib/langchain/assistant/llm/adapters/openai.rb:42
def build_message(role:, content: nil, image_url: nil, tool_calls: [], tool_call_id: nil)
Messages::OpenAIMessage.new(role: role, content: content, image_url: image_url, tool_calls: tool_calls, tool_call_id: tool_call_id)
end
That ignore the fact that the method to_hash, from the same object, transform content into an hash that merge text message and image url
This PR is extracted from a monkey patch I made in my project :
module Langchain
class Assistant
module LLM
module Adapters
class OpenAI < Base
def build_message(role:, content: nil, image_url: nil, tool_calls: [], tool_call_id: nil)
if content.is_a?(Array)
content.each do |c|
content = c[:text] if c[:type] == 'text'
image_url = c[:image_url][:url] if c[:type] == 'image_url'
end
end
Messages::OpenAIMessage.new(
role: role,
content: content,
image_url: image_url,
tool_calls: tool_calls,
tool_call_id: tool_call_id
)
end
end
end
end
end
end
I also added tests for the #build_message that covers the previous behavior and the one added