PromptingTools.jl
PromptingTools.jl copied to clipboard
[FR] Add support for structured extraction with Ollama models
It would be good to have aiextract
enabled for Ollama models.
There is no official support yet, but you can easily build it yourself with the following guide: https://svilupp.github.io/PromptingTools.jl/dev/how_it_works#Walkthrough-Example-for-aiextract
It works well with mixtral and similar models.
I'm willing to handle this one, relatively straightforward to do. As a clarifying question, what's the difference between a regular schema and a managed one?
Great!
There are different API endpoints in Ollama:
- generate - basically text completion-like; where you provide two fields: system, prompt; no multi-turn conversation (1 reply only)
- api/chat - "message"-based with multiturn conversations. Similar to OpenAI-style
- v1/chat/completions - fully OpenAI-compatible endpoint
- generate is the OllamaManagedSchema, because they manage everything. It's legacy and I didn't want to break things to much so I kept it
- api/chat is the OllamaSchema
- not implemented because there was no need / no advantage (at the time)
From that perspective, I'd assume you would add aiextract for OllamaSchema which is build around api/chat
. You would just merge the api_kwargs to also include the format="json"
and then try to convert to the return type in a try-catch block. Similar to the OpenAI implementation.
I'd suggest to avoid nested return_types (they are harder for OSS models). Good model to use for aiextract is mixtral if you can run it locally.
Does that answer your question?
Just flagging that it might be easier to tackle this after we can provide "JSON type" representation to the open-source models (not JSON schema) - reference.
See https://github.com/svilupp/PromptingTools.jl/issues/143