John Beatty

icon indicating a website link https://www.johndbeatty.com icon indicating an email [email protected]

icon company Bench Computing, Inc. icon location Bay Area/Peninsula icon location Engineer, Entrepreneur. Co-founder, Bench Computing. Former co-founder/CEO of Clover.

Results 2 issues of John Beatty

Randomly getting truncated output from client.run() for streaming models using replicate 0.24.0

1 comment

With replicate 0.24.0 Python client and "mistralai/mistral-7b-instruct-v0.2" (which is a model that supports streaming), the iterator I get back from client.run() is truncating output frequently, perhaps 1/50 times. I checked...

Support logprobs, max_logprobs, and logit_bias params to match OpenAI's recent support for them in Chat Completions

See https://twitter.com/OpenAIDevs/status/1735730662362189872 and https://platform.openai.com/docs/api-reference/chat/create