cortex.cpp icon indicating copy to clipboard operation
cortex.cpp copied to clipboard

Published 20 hours ago • verified publisher icon janhq

feat: Support `response_format` and structured JSON responses.

Open actow opened this issue 8 months ago • 2 comments

  • [X] I have searched the existing issues

Is your feature request related to a problem? Please describe it

There is not a away to force the model to return structured json at the API level.

Describe the solution

The response_format parameter is supported by certain models, such as Groq's llama3 (8b & 70b), Fireworks AI's llama3 70b and OpenAI gpt 3.5 and 4 etc.

https://console.groq.com/docs/api-reference#chat-create https://platform.openai.com/docs/api-reference/audio/createSpeech#audio-createspeech-response_format https://readme.fireworks.ai/docs/structured-response-formatting

If will be great if Jan provide a UI to set that.

Teachability, documentation, adoption, migration strategy

https://console.groq.com/docs/api-reference#chat-create https://platform.openai.com/docs/api-reference/audio/createSpeech#audio-createspeech-response_format https://readme.fireworks.ai/docs/structured-response-formatting

What is the motivation / use case for changing the behavior?

The JSON structure response is very useful to experiment the models' capability to extend beyond a normal chat bot.

actow avatar Jun 24 '24 05:06 actow