Add support for GBNF grammar definitions

[SyrupThinker]

This PR exposes the llama.cpp grammar parameter in the generate API.

It allows the user to provide a GBNF grammar to constrain the output of an LLM.

This can be used to, for example, reliably generate structured data like JSON:

>>> Generate a list of 5 random mock users that contain a firstname, lastname, birthday, created_at and email field. The created_at field should be RFC3339 and lie in the range of 2000 to 2020. Emails should use multiple subdomains under the example.com domain. The result should be in a JSON object with a users key.
{
    "users": [
      {
        "firstname": "Emma",
        "lastname": "Brown",
        "birthday": "1993-08-12T00:00:00Z",
        "created_at": "2017-04-15T13:00:00Z",
        "email": "[email protected]"
      },
      {
        "firstname": "Olivia",
        "lastname": "Jones",
        "birthday": "1996-03-25T00:00:00Z",
        "created_at": "2018-02-17T14:00:00Z",
        "email": "[email protected]"
      },
      {
        "firstname": "Ava",
        "lastname": "Smith",
        "birthday": "1997-08-24T00:00:00Z",
        "created_at": "2019-05-12T15:00:00Z",
        "email": "[email protected]"
      },
      {
        "firstname": "Sophia",
        "lastname": "Johnson",
        "birthday": "1998-04-26T00:00:00Z",
        "created_at": "2020-03-07T16:00:00Z",
        "email": "[email protected]"
      },
      {
        "firstname": "Mia",
        "lastname": "Williams",
        "birthday": "1999-07-23T00:00:00Z",
        "created_at": "2020-12-08T17:00:00Z",
        "email": "[email protected]"
      }
    ]
  }

>>> Create a JSON object that contains the latest birthday and the earliest created_at date, omit the time.
{
"latest_birthday": "1999-07-23",
"earliest_created_at": "2000-01-01"
}

Generated with the examples/json Modelfile, first attempt, not cherry-picked

A note for potential users The generated documents are first try valid JSON, without extra tuning. But note how the LLM used different TLD's, not subdomains. The earliest_created_at is also not as intended, the instruction is ambiguous. This only ensures that the grammar is followed, the semantics might still be wrong.

[clevcode]

This is such an obviously very useful PR + it's such a small and simple patch. How come it has not been merged yet?

[clevcode]

When looking at the current code, I see that support has been added specifically for grammars conforming to JSON which are then translated into the corresponding GGBNF, and although that's certainly a welcome addition, there are tons of use-cases where you want to actually be able to enforce an arbitrary grammar.

There's a small conflict with this patch and the current repo. I will fix that and try creating another PR. Hopefully it gets merged

[SyrupThinker]

When looking at the current code, I see that support has been added specifically for grammars conforming to JSON which are then translated into the corresponding GGBNF, and although that's certainly a welcome addition, there are tons of use-cases where you want to actually be able to enforce an arbitrary grammar.

I'm not sure what you mean by this, the PR exposes the llama.cpp grammar parameter as is, no translation or similar. The Modelfile containing the JSON grammar is merely an example and could be removed.

There's a small conflict with this patch and the current repo. I will fix that and try creating another PR. Hopefully it gets merged

No worries, I can quickly fix that up considering that there is interest for the change.

[SyrupThinker]

@BruceMacD Apologies for the unsolicited ping, especially this close to the holidays. Doubly so if you're not the right person to ping for this, I based it on recent merges.

I was just wondering like clevcode if there is something I can do to get this along, it seems like there is a decent amount of interest for this change.

[jacopofar]

This would be really useful. There are tools to generate a grammar for a specific JSON schema and it could be used to enforce it. If I understand correctly the current implementation can enforce a valid JSON as output but does not enforce a particular structure on it so it's not really machine readable.

[UmutAlihan]

Please merge this PR

[hchasens]

@brucemacd Any update on this?

[SyrupThinker]

Given the recent release of tool support I have basically no need for this feature anymore. So I'll close this PR in favor of one of the many follow up ones that do the same thing.

[recreationalviking]

I'd still like to see this merged =(

There are so many uses for arbitrary grammars.

[jacopofar]

Agree, it's quite an essential feature and it seems "just" a matter of propagating the grammar to the model.

I'm not sure in general tools do enforce a grammar or just rely on the fact the LLM is going to generate a JSOn with the appropriate structure.

[Kinglord]

Hey all, I know there's an automated ping here but just to better align everyone please check out and comment on my new call to the Ollama team for clarity here. As always please be civil and stay on topic! 😄 - https://github.com/ollama/ollama/issues/6237