tensorzero icon indicating copy to clipboard operation
tensorzero copied to clipboard

Best-of-N sampling doesn't correctly handle JSON mode for Anthropic models

Open virajmehta opened this issue 1 year ago • 2 comments

3 options to fix:

  1. add a check that if the model being called is Anthropic then we use ImplicitTool.
  2. Make JSON model default to ImplicitTool for Anthropic
  3. Add the Anthropic prefill thing to the implementation as in #133

virajmehta avatar Sep 19 '24 15:09 virajmehta

Let's solve via #133

GabrielBianconi avatar Oct 11 '24 21:10 GabrielBianconi

Anthropic JSON mode enforcement is flaky and in particular has an issue where it generates multiline strings. This is an issue generally and in particular for "thinking" blocks. For now I don't think this is solvable. Consider deprioritizing.

virajmehta avatar Oct 14 '24 15:10 virajmehta

If this is not actionable, I'll close for now and we'll hope that Anthropic adds more JSON capabilities. It's already better than before with TensorZero's implicit tool use.

GabrielBianconi avatar Dec 29 '24 16:12 GabrielBianconi