ChatMessage content being `str`-only doesn't allow user to pass image

[tomarharsh]

Is your feature request related to a problem? Please describe. While talking to our bot, the user is allowed to send an image. This image is sent to vision enabled LLM bot. Haystack ChatMessage class content only allows string but it needs to allow a List to be passed. Here's the OpenAI page the Haystack refers to for content which allows array and image_url that can be sent that way.

Describe the solution you'd like ChatMessage to be able to handle inbound image

Describe alternatives you've considered Not using generator component at all is the only other alternative I can explore.

Additional context Haystack's ChatMessage content: Link OpenAI's chat message parameter: Link How ChatMessage content is getting populated from the generator: Link

[CarlosFerLo]

I will try to add this functionality :)

[CarlosFerLo]

I've reviewed the base code and propose that we enable the 'content' of a 'ChatMessage' to be set as a list containing 'str', 'Path', or any type used to encode an image. This will require us to rewrite the 'to_openai_format' method and incorporate image processing with 'base64' for calls involving images. We'll also need to address serialization issues, but we can handle those once #7849 is merged into the main branch to avoid merge conflicts.

The main challenge will be accurately distinguishing between images and text in the input list, especially when the input is a string. It would be helpful to know which data types you want to support for images. I'll begin working on this after the mentioned PR is merged.

[lbux]

The main challenge will be accurately distinguishing between images and text in the input list, especially when the input is a string. It would be helpful to know which data types you want to support for images.

I don't think we should try and extract this info ourselves. We should make the user specify. My idea is to make a ContentPart class with type, text, image_url, base_64, and detail. We can then have helper methods in this class that helps with formatting.

Essentially, we would allow for something like this:

message = ChatMessage.from_user([
    ContentPart.from_text("What’s in this image?"),
    ContentPart.from_image_url("example.com/test.jpg"),
    ContentPart.from_base64_image(base64_image)
])

We should also look into deprecating Functions and supporting Tools within ChatMessage as that has also changed.

[CarlosFerLo]

I will implement this functionality. Regarding the deprecation of Functions, we could open an issue to handle it separately.

[vblagoje]

The main challenge will be accurately distinguishing between images and text in the input list, especially when the input is a string. It would be helpful to know which data types you want to support for images.

I don't think we should try and extract this info ourselves. We should make the user specify. My idea is to make a ContentPart class with type, text, image_url, base_64, and detail. We can then have helper methods in this class that helps with formatting.

Essentially, we would allow for something like this:

message = ChatMessage.from_user([
    ContentPart.from_text("What’s in this image?"),
    ContentPart.from_image_url("example.com/test.jpg"),
    ContentPart.from_base64_image(base64_image)
])

We should also look into deprecating Functions and supporting Tools within ChatMessage as that has also changed.

I agree with this direction. We need to look at all the multimodal message formats across all LLM providers and deduce common denominators. From a brief cursory look I believe these multimodal/multipart messages are all json payloads of various formats (schemas). So let's come up with a nice abstractions (like the ContentPart idea above) that abstracts the implementation details and see how they map to data structures across various LLM providers.

[silvanocerza]

We can keep it much simpler.

As of now models can receive and generate the following:

• text
• image
• audio
• video
• heterogeneous list of all the above

We have all the necessary abstractions to define the above. str obviously for text. haystack.dataclasses.ByteStream for image, audio and video. The list is List[Union[str, ByteStream]] then.

Given that we say that ChatMessage.content type should be Union[str, ByteStream, List[Union[str, ByteStream]]].

This abstracts at an high level all the supported type of data a model receives and generates. If model X needs their input or generates their output in a certain format its Generator will handle the conversion, but that's an implementation detail.

Introducing new classes or new abstractions is not the way to go in my opinion.

[CarlosFerLo]

@silvanocerza I like the simplicity of your solution, but I've just read the code for 'ByteStream' and we should expect the metadata to be populated with some flag to indicate the content type, else we won't be able to distinguish. That's why I believe that the 'ContentPart' approach to be easier to handle and allows us to provide for brother input types for the different formats. I will proceed with this implementation as soon as #7849 is merged to main.

[silvanocerza]

Still relevant, reopening.

[vblagoje]

Still relevant, reopening.

I agree - schedule it soon as well.

[lbux]

We can keep it much simpler.

As of now models can receive and generate the following:

• text
• image
• audio
• video
• heterogeneous list of all the above

We have all the necessary abstractions to define the above. str obviously for text. haystack.dataclasses.ByteStream for image, audio and video. The list is List[Union[str, ByteStream]] then.

Given that we say that ChatMessage.content type should be Union[str, ByteStream, List[Union[str, ByteStream]]].

This abstracts at an high level all the supported type of data a model receives and generates. If model X needs their input or generates their output in a certain format its Generator will handle the conversion, but that's an implementation detail.

Introducing new classes or new abstractions is not the way to go in my opinion.

Trying to understand how this implementation would work. Let's say that the ChatMessage class is modified to change content to be Union[str, ByteStream, List[Union[str, ByteStream]]]. Given this, we should be able to pass in a ByteStream to ChatMessage. This, I understand. However, my understanding is that you want to move the actual implementation to each generator?

Then does that mean we would need to handle the conversion in, say _convert_message_to_openai_format, like so:

def _convert_message_to_openai_format(message: ChatMessage) -> Dict[str, str]:
    """
    Convert a message to the format expected by OpenAI's Chat API.

    See the [API reference](https://platform.openai.com/docs/api-reference/chat/create) for details.

    :returns: A dictionary with the following key:
        - `role`
        - `content`
        - `name` (optional)
    """

    openai_msg = {"role": message.role.value}

    if isinstance(message.content, str):
        openai_msg["content"] = message.content
    elif isinstance(message.content, ByteStream):
        base64_data = b64encode(message.content.data).decode("utf-8")
        url = f"data:{message.content.mime_type};base64,{base64_data}"
        openai_msg["content"] = [({"type": "image_url", "image_url": {"url": url}})]
    elif isinstance(message.content, list):
        openai_msg["content"] = []
        for item in message.content:
            if isinstance(item, str):
                openai_msg["content"].append({"type": "text", "text": item})
            elif isinstance(item, ByteStream):
                base64_data = b64encode(item.data).decode("utf-8")
                url = f"data:{item.mime_type};base64,{base64_data}"
                openai_msg["content"].append(({"type": "image_url", "image_url": {"url": url}}))

    if message.name:
        openai_msg["name"] = message.name

    return openai_msg

This works provided that the user specifies the valid mime_type in ByteStream.from_file_path (or we can try to infer it like so:

def from_file_path(
        cls, filepath: Path, mime_type: Optional[str] = None, meta: Optional[Dict[str, Any]] = None
    ) -> "ByteStream":
        """
        Create a ByteStream from the contents read from a file.

        :param filepath: A valid path to a file.
        :param mime_type: The mime type of the file.
        :param meta: Additional metadata to be stored with the ByteStream.
        """

        if mime_type is None:
            mime_type = mimetypes.guess_type(filepath)[0]
            if mime_type is None:
                raise ValueError("Mime type was not supplied and could not be guessed.")
        
        with open(filepath, "rb") as fd:
            return cls(data=fd.read(), mime_type=mime_type, meta=meta or {})

With this implementation, the abstractions aren't modified as much and the conversions would (probably) occur in some helper functions for each generator. This would allow for usage like so:

message = [ChatMessage.from_user(content=["Write me a poem about this image", ByteStream.from_file_path("nier.jpg")])]
generator = OpenAIChatGenerator(api_key = Secret.from_env_var("OPENAI_API_KEY"), model = "gpt-4o-mini")
output = generator.run(messages=message)

Is this what you had in mind or is there some other insight you could provide to help with an implementation?

[joshdawson]

Just wondering if this is any further along or if there are any new workarounds to feed images in to a chat generator?

[Jchang4]

Yes please make this higher priority! This is a huge piece of functionality from Chat GPT

[lbux]

Trying to understand how this implementation would work. Let's say that the ChatMessage class is modified to change content to be Union[str, ByteStream, List[Union[str, ByteStream]]]. Given this, we should be able to pass in a ByteStream to ChatMessage. This, I understand. However, my understanding is that you want to move the actual implementation to each generator?

With this implementation, the abstractions aren't modified as much and the conversions would (probably) occur in some helper functions for each generator. This would allow for usage like so:

message = [ChatMessage.from_user(content=["Write me a poem about this image", ByteStream.from_file_path("nier.jpg")])]
generator = OpenAIChatGenerator(api_key = Secret.from_env_var("OPENAI_API_KEY"), model = "gpt-4o-mini")
output = generator.run(messages=message)

Is this what you had in mind or is there some other insight you could provide to help with an implementation?

Ollama just released vision support. If we stick to the bytestream implementation I suggested, we can add support to it in the Ollama implementation by doing something like

def _message_to_dict(self, message: ChatMessage) -> Dict[str, Union[str, List[str]]]:
        result = {"role": message.role.value}

        # Handle content field
        if isinstance(message.content, str):
            result["content"] = message.content
        elif isinstance(message.content, list):
            # Concatenate text in list and handle images
            text_content = []
            images = []
            for item in message.content:
                if isinstance(item, str):
                    text_content.append(item)
                elif isinstance(item, ByteStream):
                    base64_data = b64encode(item.data).decode("utf-8")
                    images.append(base64_data)
            result["content"] = " ".join(text_content)
            if images:
                result["images"] = images
        elif isinstance(message.content, ByteStream):
            base64_data = b64encode(message.content.data).decode("utf-8")
            result["content"] = ""
            result["images"] = [base64_data]
        else:
            result["content"] = ""

        return result

Which would once again look the same as the OpenAI implementation when the user calls it.

message = [ChatMessage.from_system(content="Talk like a pirate"), ChatMessage.from_user(content=["Write me a poem about this image.", ByteStream.from_file_path("nier.jpg")])]
generator = OllamaChatGenerator(model="llama3.2-vision")
output = generator.run(messages=message)

[LastRemote]

Hi @silvanocerza , do you mind sharing more about the broader vision for ChatMessage? Besides the multimodal support discussed here, I've noticed some recent updates related to tool calls in the experimental repo. These have been around for a while, and I'm curious about whether transitioning to the new architecture is recommended. Additionally, could you outline the roadmap or overall strategy for rolling out these features into production?

[LastRemote]

I am planning to add multimodal support in haystack-experimental (it already has some advanced tool supports there). I am opening an issue for this (basically my to-do list for better visibility since I am aware that some people are asking multimodality): https://github.com/deepset-ai/haystack-experimental/issues/135