evals
evals copied to clipboard
openai
GPT 4 Image Evals
Hello, I wanted to open an issue/question regarding GPT-4 evals with images. I have a few questions as I'm looking to write some evals to test a few ideas I had.
- Is there a rough timeline for when GPT-4 with image scanning will be available?
- When this feature comes out, will queries have to be done individually? Can we create a batch of images and include a prompt?
- If someone were to process large quantities of large images, say 10k images a day, what would that look like cost wise?
I plan to write an eval to test a significantly large dataset of high-quality 360 images of cars and answer questions the end user of a site like Carvana may have. Questions like:
- Does this car have any visible markings, dents, or damage on the surface?
- Will this car be able to take me off the road when I go camping?
- Is this a good fuel-efficient car for me to use?
- ...
Would an evaluation like this be helpful to the public? I want to test this for my needs, but if this would be useful, I can take a fragment of the set and upload it somewhere once I write the prompts.
I'm curious; I've reached out to the OpenAI team but haven't heard back. My guess is they won't have pricing for a while, it seems like getting GPT-4 to scale for more users is the current challenge.
I think one of the issues is the availability of the Nvidia chips needed to power the platform.
We are interested in supporting image evals as soon as that feature becomes more widely available and the API is finalized.
@andrew-openai that's great! We are super excited for that! We're looking forward to the image feature as well as taking advantage of plugins to create tools that can interact with our vehicle databases. Hopefully this is something we can set up soon to start testing!
Going to close for now, please feel free to reopen once you start building the eval :)