Add support to include images for OpenAI Chat Template

[bhimrazy]

🚀 Feature Request: Add Support for Images in Chat Messages

Description

Add support for images to be included in chat messages, similar to gpt-4o.

{
  "messages": [
     {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                },
            ],
        }
  ]
}

Motivation

I was trying to implement a chat generation API using the phi3-vision model, which takes images as input. The proposed format will facilitate interaction with models that involve images. This approach will simplify the handling of different prompt techniques, which can be managed separately before passing the prompt to the model.

Benefits

• Enhanced Interaction: Enables richer interactions by allowing images in chat messages.
• Flexibility: Provides a structured way to handle various prompt techniques involving images.

Additional Context

Implementing this feature will involve updating the schema to support image URLs in chat messages, ensuring compatibility with existing and future vision models.

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[williamFalcon]

@bhimrazy great idea. want to take a stab at submitting a PR? we can help you finish and land it.

[bhimrazy]

@bhimrazy great idea. want to take a stab at submitting a PR? we can help you finish and land it.

Sure @williamFalcon. Sounds great.