tensorzero
tensorzero copied to clipboard
Best-of-N sampling doesn't correctly handle JSON mode for Anthropic models
3 options to fix:
- add a check that if the model being called is Anthropic then we use ImplicitTool.
- Make JSON model default to ImplicitTool for Anthropic
- Add the Anthropic prefill thing to the implementation as in #133
Let's solve via #133
Anthropic JSON mode enforcement is flaky and in particular has an issue where it generates multiline strings. This is an issue generally and in particular for "thinking" blocks. For now I don't think this is solvable. Consider deprioritizing.
If this is not actionable, I'll close for now and we'll hope that Anthropic adds more JSON capabilities. It's already better than before with TensorZero's implicit tool use.