moondream icon indicating copy to clipboard operation
moondream copied to clipboard
verified publisher icon vikhyat

Killed

Open seven7-AI opened this issue 1 year ago • 5 comments

Hello, I keep on getting this when i run the model

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Killed

How can I solve this?

seven7-AI avatar Mar 06 '24 12:03 seven7-AI

same question. did you resolved?

wfnian avatar Mar 07 '24 11:03 wfnian

Not Yet.

seven7-AI avatar Mar 07 '24 12:03 seven7-AI

Pretty sure this is the OOM killer kicking in when you run out of RAM. I saw someone mention it needs ~9.7GB of RAM available to run on CPU right now since it's not quantized and PyTorch forces us to run in 32-bit precision on CPU. Adding swap can help in the short run, but we're also working on adding support for 16-bit/quantized inference.

vikhyat avatar Mar 15 '24 01:03 vikhyat

+1 to this issue; it is currently a blocker to using this VLM on a Raspberry Pi.

A-Vaillant avatar Apr 26 '24 20:04 A-Vaillant

We have support in llama.cpp now, would recommend using that when running on Raspberry Pi. Can you please try it out and let me know if you run into any issues?

vikhyat avatar Apr 30 '24 19:04 vikhyat