langchainrb icon indicating copy to clipboard operation
langchainrb copied to clipboard

Fix Assistant OpenAI adapter to handle message content structure returned by to_hash method

Open IMhide opened this issue 8 months ago • 0 comments

While using the gem for the first time I was discovering Langchain::Assistant and experimenting with it. While using naively I came across an issue :

@assistant = Langchain::Assistant.new(
  llm: Langchain::LLM::OpenAI.new(
	api_key: ENV['OPENAI_API_KEY'],
	default_options: {
	  temperature: 0.7,
	  chat_model: 'gpt-4o-mini'
	}
  ),
  instructions: "FOO BAR ZOO",
)
@assistant.add_message(content: "Foo bar Zoo") 
messages_hash = @assistant.messages.map(&:to_hash)

Here message_hash equal

[{role: "system", content: [{type: "text", text: "FOO BAR ZOO"}]}, {role: "user", content: [{type: "text", text: "Foo bar Zoo"}]}]

From there I was thinking great I can just persist messages_hash somewhere and then use the method @assistant.add_messages to resume the conversation But this happened :

@assistant.clear_messages!
@assistant.add_messages(messages: messages_hash)
resumed_hash = @assistant.messages.map(&:to_hash)

resumed_hash equal this

[{role: "system", content: [{type: "text", text: "[{type: \"text\", text: \"FOO BAR ZOO\"}]"}]},
 {role: "user", content: [{type: "text", text: "[{type: \"text\", text: \"Foo bar Zoo\"}]"}]}]

The message_hash get stringified and nested into a new hash

After some investigation I found out that this behaviour comes from the function build_message at /lib/langchain/assistant/llm/adapters/openai.rb:42

def build_message(role:, content: nil, image_url: nil, tool_calls: [], tool_call_id: nil)
	Messages::OpenAIMessage.new(role: role, content: content, image_url: image_url, tool_calls: tool_calls, tool_call_id: tool_call_id)
end

That ignore the fact that the method to_hash, from the same object, transform content into an hash that merge text message and image url

This PR is extracted from a monkey patch I made in my project :

module Langchain
  class Assistant
    module LLM
      module Adapters
        class OpenAI < Base
          def build_message(role:, content: nil, image_url: nil, tool_calls: [], tool_call_id: nil)
            if content.is_a?(Array)
              content.each do |c|
                content = c[:text] if c[:type] == 'text'
                image_url = c[:image_url][:url] if c[:type] == 'image_url'
              end
            end
            Messages::OpenAIMessage.new(
              role: role,
              content: content,
              image_url: image_url,
              tool_calls: tool_calls,
              tool_call_id: tool_call_id
            )
          end
        end
      end
    end
  end
end

I also added tests for the #build_message that covers the previous behavior and the one added

IMhide avatar Apr 12 '25 20:04 IMhide