ollama MLX backend

MLX backend

Open ageorgios opened this issue 6 months ago • 6 comments

Can ollama be converted to use MLX from Apple as backend for the models ?

Dec 27 '23 20:12 ageorgios

This Please!

Dec 31 '23 09:12 Josecodesalot

What do you hope to gain from this? I don't think MLX is faster for inference, at least not yet.

Jan 02 '24 20:01 easp

Found these benchmarks: https://medium.com/@andreask_75652/benchmarking-apples-mlx-vs-llama-cpp-bbbebdc18416

Seems like MLX is indeed slower than the llama.cpp masterpiece, at least for now. I did not verify though.

Jan 10 '24 04:01 KernelBypass

This would be very nice! and not only for text generation, Image/Multimodal would be boosted too.

Jan 23 '24 03:01 Edu126

someone made this https://github.com/kspviswa/PyOMlx

Apr 20 '24 12:04 JimmyLv

Ollama is awesome and does so many things and some of us want to play with mlx models.

May 04 '24 04:05 magnusviri