Convert to Gguf format to work with Llama.cpp?

[chigkim]

Llava has various quantized models in gguf format, so it can be used with Llama.cpp. https://github.com/ggerganov/llama.cpp/pull/3436 Is this possible?

[czczup]

Hi, thank you for your suggestion. I will add compatibility with community tools to my to-do list.

[leeaction]

gguf format is good for ollama users, Any Update?

[GHOST1834]

It will be nice to have this model in gguf format in ollama.

[nischalj10]

any updates on this? the 4b intern model is killer for its size! would love to see it supported with llama.cpp

[KOG-Nisse]

Would love internvl-chat-v1-5 in a gguf format! https://internvl.opengvlab.com/

[thomas-rooty]

I second this

[orabazes]

@ErfeiCui why did you close this as completed?

[chigkim]

Any update on this? InternVL2-Llama3-76B on Ollama/llama.cpp would be amazing!

[kim-gtek]

If someone gives me a tutorial I will write my own code to tranform this for pytorch to gguf for llama.cpp myself

[chigkim]

It's more involved. you have to implement the model architecture and image preprocessing logic to llama.cpp which uses C++.