julep
julep copied to clipboard
[Bug]: Remove support for `openai.chat.completions.create(..., n>1)` everywhere
trafficstars
📜 Description
So the problem is that if you use n>1, api will return more than one result in the {"choices": [...]} but then:
- how do you unwrap?
- if there's a tool-call, how do you handle it as in which choice's tool-call should you use?
- no one really uses choices anymore (reasoning models don't support it either)
Possible solutions
- Remove support from typespec and rewrite (simplify) the prompt step logic
- Just consider tool-calls with
n>1as wait for input
👀 Have you spent some time to check if this bug has been raised before?
- [X] I checked and didn't find similar issue
@whiterabbit1983 thoughts?