Unable to send BinaryPart to OpenAI Completions API

[pilsnerbeer]

Trying to directly send a image/png to OpenAI completions API. (Model: Gpt-4o / Mini)

Snippet:

		if imageData != nil {
			historyWithBinary = append(historyWithBinary, llms.MessageContent{
				Role: llms.ChatMessageTypeHuman,
				Parts: []llms.ContentPart{
					llms.BinaryPart("image/png", imageData),
				},
			})
		}

		choices, err := client.GenerateContent(context.Background(), historyWithBinary, llms.WithTools([]llms.Tool{FileWriteTool}))

Returns Error:

Error generating content: API returned unexpected status code: 400: Invalid value:
'binary'. Supported values are: 'text', 'image_ url', and 'audio_url'.

This seems to be only problematic with OpenAI. Gemini and Ollama worked fine when i tested it with the same snippet. According to https://platform.openai.com/docs/guides/vision doc sending images directly to vision API shoudl be possible so the error is not clear to me

[ccrlawrence]

I have this same issue, I can workaround with:

base64Image := base64.StdEncoding.EncodeToString(jpegBytes)

then llms.ImageURLPart(fmt.Sprintf("data:image/jpeg;base64,%s", base64Image))

Not sure if BinaryPart should just be supported anyway? Ideally it would somehow?

[filintod]

The api link does not show any mention of binary format, so what @ccrlawrence mentioned above is the way to go

[iamalsaher]

Bumping this for, what if I dont want to sent image files, and other files like pdf files

[23RoMax]

Also bumping this for the same reason. We are working with PDFs.

[serbanmarti]

It would be great if we could get that PR in and released. This is an annoying problem to have to deal with.