langchaingo icon indicating copy to clipboard operation
langchaingo copied to clipboard

[feat] googleai file api support

Open douglarek opened this issue 1 year ago • 2 comments

link: https://ai.google.dev/gemini-api/docs/prompting_with_media?lang=go .

douglarek avatar Jun 24 '24 06:06 douglarek

What is the general cross-provider interface this would be behind?

eliben avatar Jun 24 '24 21:06 eliben

I propose adding a new ContentPart type called FileURLContent to represent generic file URIs with associated MIME types.

Some existing types:

type BinaryContent struct {
    MIMEType string
    Data     []byte
}

type ImageURLContent struct {
    URL    string `json:"url"`
    Detail string `json:"detail,omitempty"` // Detail of the image, e.g., "low", "high".
}

Proposed addition:

type FileURLContent struct {
    MIMEType string
    URL      string
}

For example, for Google AI, we can add another case in convertParts for genai.FileData:

case llms.FileURLContent:
    out = genai.FileData{
        MIMEType: p.MIMEType,
        FileURI:  p.URL,
    }
}

nemo984 avatar Oct 14 '24 05:10 nemo984