Feature Request: Req to support Structured output and JSON schema to GBNF

[jkfnc]

Prerequisites

• [X] I am running the latest code. Mention the version if possible as well.
• [X] I carefully followed the README.md.
• [X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [X] I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Openai updated sdk with support for structured output and json schema. Is it possible to get similar feature added to completion api but compiling to GBNF grammar. https://platform.openai.com/docs/guides/structured-outputs

Motivation

This will enable feature parity with completions api.

Possible Implementation

No response

[James4Ever0]

Made one proposal to solve this problem:

https://github.com/ollama/ollama/issues/6302

[webitube]

Llamasharp already supports GBNF grammars. I've been using them for a while now. Take a look at the LLama.Unittest/GrammarTest.cs for example usage. It's a little convoluted, but it works. You have to create a SafeLLamaGrammarHandle for your grammar and pass that handle in your InferenceParams. Just make sure you use a Clone() of the handle instead of the original handle. Otherwise, it would be disposed afterwards.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.