Deploying olmOCR-7B-0225-preview-GGUF using Ollama, how to call it with OpenAI

Open netty2019 opened this issue 9 months ago • 4 comments

🚀 The feature, motivation and pitch

Alternatives

No response

Additional context

No response

Feb 27 '25 01:02 netty2019

The model is based on https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct so it may not be supported by Ollama at this time. Try VLLM or SGlang, which is what we tested on.

Feb 27 '25 04:02 jakep-allenai

https://github.com/ollama/ollama/issues/6564

Mar 03 '25 10:03 James4Ever0

The model is based on https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct so it may not be supported by Ollama at this time. Try VLLM or SGlang, which is what we tested on.

Hello, how is it going?

Mar 10 '25 11:03 lovebaby

Ollama still does not work and llama.cpp on q8 is bad. Use vllm. https://github.com/allenai/olmocr/issues/347

Oct 16 '25 09:10 montvid