Error with multi-modal chat and agent memory

[nacartwright]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.

Example Code

    llm = BedrockChat(
        credentials_profile_name="default",
        model_id="anthropic.claude-instant-v1",
        streaming=True,
        model_kwargs={"temperature": 0.1},
    )

    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            MessagesPlaceholder(variable_name="history"),
            ("human", "{input}"),
            MessagesPlaceholder(variable_name="agent_scratchpad"),
        ]
    )

    tools = load_tools(["google-search"], llm)
    image_generation_tool = StructuredTool.from_function(
        func=image_generation,
        name="image_generator",
        description="Use this tool to generate images for the user",
        return_direct=True,
    )

    tools.append(image_generation_tool)

    agent = create_json_chat_agent(llm, tools, prompt)

    history = DynamoDBChatMessageHistory(table_name="LangchainSessionTable", session_id=session_id)
    memory = ConversationBufferMemory(chat_memory=history, memory_key="history", return_messages=True)

    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True, memory=memory)```

### Error Message and Stack Trace (if applicable)


> Finished chain.
2 validation errors for AIMessage
content
  str type expected (type=type_error.str)
content
  value is not a valid list (type=type_error.list)

### Description

I'm using:

1. Langchain 0.5.1
2. Amazon Bedrock / Anthropic Claude Instant 1.2
3. Amazon Dynomo DB for chat history
4. Conversation memory buffer
5. A tool to create an image from Bedrock Stability AI

The agent generates the image but when it tries to add it to the conversation history; I get an error. 

### System Info


System Information
------------------
> OS:  Linux
> OS Version:  #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2
> Python Version:  3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]

Package Information
-------------------
> langchain_core: 0.1.21
> langchain: 0.1.5
> langchain_community: 0.0.19
> langsmith: 0.0.87
> langchain_openai: 0.0.5
> langserve: 0.0.41

Packages not installed (Not Necessarily a Problem)
--------------------------------------------------
The following packages were not found:

> langgraph

[dosubot[bot]]

🤖

Hey there, @nacartwright! Great to see you back here. Hope all's well on your end.

Based on the error message you provided, it seems like the issue is related to the AIMessage content. The error message indicates that a string type was expected for the content, but a list was provided instead. This could be due to the image data being passed as a list instead of a string.

The AIMessage class in LangChain represents a message from an AI and it expects the content to be a string. If you're trying to pass image data into the AIMessage, it might be causing the error because the image data is not a string.

To resolve this issue, you could convert the image data to a string before passing it into the AIMessage. Here's an example of how you might do this:

from PIL import Image
import base64
from io import BytesIO

# Load the image
img = Image.open('image.png')

# Convert the image to base64
buffered = BytesIO()
img.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode()

# Now you can pass img_str into the AIMessage
message = AIMessage(content=img_str)

In this example, the image is loaded using the PIL library, then it's converted to a base64 string using the base64 library. This string can then be passed into the AIMessage.

Please note that this is a simplified example and you might need to adjust it according to your specific requirements. For instance, you might need to handle different image formats, or you might need to include additional information in the AIMessage.

If this doesn't resolve your issue, could you please provide more details about how you're generating the image and adding it to the conversation history? This would help in identifying the exact cause of the issue.

Sources

• libs/core/langchain_core/chat_history.py
• libs/core/langchain_core/messages/ai.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nacartwright]

The image is already a base64 string

[LatVAlY]

I got the same issue pydantic.v1.error_wrappers.ValidationError: 2 validation errors for AIMessage