instructor_ex icon indicating copy to clipboard operation
instructor_ex copied to clipboard

[Gemini adapter] Support message content with image and video URLs

Open samrat opened this issue 1 year ago • 2 comments

Fixes #80

The Gemini API expects you to upload images/videos separately. That feels outside of the scope of Instructor, so I've avoided adding that in this PR.

Example usage:

defmodule VideoDesc do
  use Ecto.Schema

  @primary_key false
  embedded_schema do
    field(:video_description, :string)
    field(:oscar_winning_moment, :boolean)
  end
end

Instructor.chat_completion(
  mode: :json_schema,
  model: "gemini-1.5-flash",
  response_model: VideoDesc,
  messages: [
    %{
      role: "user", 
      content: [
        %{
          type: "video_url",
          video_url: %{
            url: "https://generativelanguage.googleapis.com/v1beta/files/d2ngu357dmfx",
            mime_type: "video/mp4"
          }
        },
        %{
          type: "text",
          text: "What's going on in this video?"
        }
      ]
    }
  ]
)

samrat avatar Nov 12 '24 21:11 samrat

The Gemini API expects you to upload images/videos separately

In case anyone wants to grab the code to do the upload, I've posted it here.

samrat avatar Nov 12 '24 22:11 samrat

ty! this looks great

thmsmlr avatar Feb 09 '25 20:02 thmsmlr