instructor_ex
instructor_ex copied to clipboard
[Gemini adapter] Support message content with image and video URLs
Fixes #80
The Gemini API expects you to upload images/videos separately. That feels outside of the scope of Instructor, so I've avoided adding that in this PR.
Example usage:
defmodule VideoDesc do
use Ecto.Schema
@primary_key false
embedded_schema do
field(:video_description, :string)
field(:oscar_winning_moment, :boolean)
end
end
Instructor.chat_completion(
mode: :json_schema,
model: "gemini-1.5-flash",
response_model: VideoDesc,
messages: [
%{
role: "user",
content: [
%{
type: "video_url",
video_url: %{
url: "https://generativelanguage.googleapis.com/v1beta/files/d2ngu357dmfx",
mime_type: "video/mp4"
}
},
%{
type: "text",
text: "What's going on in this video?"
}
]
}
]
)
The Gemini API expects you to upload images/videos separately
In case anyone wants to grab the code to do the upload, I've posted it here.
ty! this looks great