[FR] Add support for structured extraction with Ollama models

[svilupp]

It would be good to have aiextract enabled for Ollama models.

[svilupp]

There is no official support yet, but you can easily build it yourself with the following guide: https://svilupp.github.io/PromptingTools.jl/dev/how_it_works#Walkthrough-Example-for-aiextract

It works well with mixtral and similar models.

[cpfiffer]

I'm willing to handle this one, relatively straightforward to do. As a clarifying question, what's the difference between a regular schema and a managed one?

[svilupp]

Great!

There are different API endpoints in Ollama:

• generate - basically text completion-like; where you provide two fields: system, prompt; no multi-turn conversation (1 reply only)
• api/chat - "message"-based with multiturn conversations. Similar to OpenAI-style
• v1/chat/completions - fully OpenAI-compatible endpoint

1. generate is the OllamaManagedSchema, because they manage everything. It's legacy and I didn't want to break things to much so I kept it
2. api/chat is the OllamaSchema
3. not implemented because there was no need / no advantage (at the time)

From that perspective, I'd assume you would add aiextract for OllamaSchema which is build around api/chat. You would just merge the api_kwargs to also include the format="json" and then try to convert to the return type in a try-catch block. Similar to the OpenAI implementation.

I'd suggest to avoid nested return_types (they are harder for OSS models). Good model to use for aiextract is mixtral if you can run it locally.

Does that answer your question?

[svilupp]

Just flagging that it might be easier to tackle this after we can provide "JSON type" representation to the open-source models (not JSON schema) - reference.

See https://github.com/svilupp/PromptingTools.jl/issues/143