Add GPT-4 Vision API wrapper

[jmanhype]

Introduce a new GPT4Vision class in visionopenai.py that wraps the GPT-4 Vision API. This abstraction layer simplifies the process of making requests to the API for analyzing images. Key functionality includes:

Encoding images to base64 Preparing image metadata in the required format Calculating token costs for images Making API requests with both text and image prompts Error handling and validation

[isaacbmiller]

Also is there any way to add tests for this?

[jmanhype]

Also is there any way to add tests for this?

I have a unittest created.

[isaacbmiller]

I have a unittest created.

Where?

[landworth]

Adding GPT-4 Vision capabilties would be super useful! Any progress with fixing the failing check here? 🙂

[nqyy]

Did you try whether this really works or not?

What does the Signature look like to pass an image?

[arnavsinghvi11]

Thanks @jmanhype for the contribution! Could you resolve https://github.com/stanfordnlp/dspy/pull/682#pullrequestreview-1950064068?

Also, it would be great if you add documentation for this under the documentation folder as with the other LMs. Thanks!

[jmanhype]

I just made those updates

[arnavsinghvi11]

@jmanhype

Also run "ruff check . --fix-only"

add documentation for this under the documentation folder as with the other LMs. Thanks!

[jmanhype]

Done

[arnavsinghvi11]

@jmanhype did you push? don't see new commits.

[jmanhype]

ok everything has been updated

[arnavsinghvi11]

Hi @jmanhype still needs ruff check . --fix-only to be run.

Also, to properly call dspy.GPT4Vision, you need to import it as the other LM modules:

• dsp module initialization
• dspy module initialization

[jmanhype]

Ruff check is false positive it passes locally

[AyushPibit1234]

@arnavsinghvi11 Is there any update on this? cc : @jmanhype

[jmanhype]

Just needs to be merged the ruff error is a false positive

[isaacbmiller]

Hey @jmanhype I did actually find a ruff error and I added GPT4Vision to the init files in dsp and dspy like @arnavsinghvi11 mentioned.

See this branch: https://github.com/stanfordnlp/dspy/tree/jmanhype/main