spring-ai
spring-ai copied to clipboard
spring-projects
Add support for GPT-4 with Vision
Having experimented with OpenAI's GPT-4 with Vision API, it would be amazing if Spring AI adds support for image-based input data (e.g. photos). This API allows you to post:
- One or more images
- Images are represented as base64 encoded data or as image URL
The API is located at: API reference: https://platform.openai.com/docs/guides/vision
An implementation could possibly also provide additional pre-processing in order to add support for video data as described here:
https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding
It looks like this feature would depend on https://github.com/TheoKanning/openai-java/issues/397
We have moved away from depending on that library and will provide our own implementations. ATM, brainstorming on having the API incorporate more than just chat models. Will be able to share a spike branch shortly. Was just circling back to issues filed wrt to other AI models. And happy new year Gunnar!
How is the spike branch going? Is there a way others could help contribute?
It should be possible to close this issue since multimodality support has been introduced in Spring AI 1.0.0-M1, including the possibility to pass images as input to OpenAI: https://docs.spring.io/spring-ai/reference/api/multimodality.html
Example: https://github.com/ThomasVitale/llm-apps-java-spring-ai/tree/main/02-prompts/prompts-multimodality-openai
Closing the issue. Thanks Thomas, issue pruning has been overdue, working through it now.