jxnl
Ability to draw multiple samples
Is your feature request related to a problem? Please describe. I'm trying to do something like Getting 50% (SoTA) on ARC-AGI with GPT-4o, and it requires drawing thousands of independent samples.
Describe the solution you'd like I'm basically trying to reopen https://github.com/jxnl/instructor/issues/576 since it seems it was closed together with the pull request https://github.com/jxnl/instructor/pull/578 because it wasn't clear "if people really needed this".
This means instructor should support the n=... argument as in:
client.chat.completions.create_with_completion(
model="gpt-3.5-turbo-0613",
response_model=Foo,
messages=[{"role": "user", "content": f"Make a python program to solve the task: {data}"},
],
n=3,
)
which would return a list[Foo] with 3 objects.
Describe alternatives you've considered Say my output type is:
class Foo(BaseModel):
solution: str
I could instead use
class Foo(BaseModel):
solutions: Annotated[list[str], Len(min_length=5, max_length=5)]
to force the model to give me 5 outputs.
However, this might overflow the output context; and it would make the outputs "dependent" when what I want are independent samples.
I could also do n independent calls to client.chat.completions.create, but when I have a very large prompt, this is very expensive.
I closed due to some breaking changes, but if it's interesting, would you be able and interested in reviving the PR.
Ok, I think you are right that Anthropic has to be excluded, as they don't support it. I guess an option would be to just do multiple calls in this case, but maybe it's better to throw an error.
Otherwise, I don't see any issues with your PR!