Model conversion from HF to GGUF crashes due to lack of memory

Open philtomson opened this issue 1 year ago • 3 comments

This was for the 8B param model in the instructions - quickly ran through the 32GB RAM on my Linux PC. Is there someplace that pre-converted GGUF versions of these models might reside so that this conversion wouldn't need to be done?

Nov 21 '24 21:11 philtomson

https://huggingface.co/brunopio/Llama3-8B-1.58-100B-tokens-GGUF

Nov 22 '24 06:11 kth8

Same issue here, with my M2 Macbook Air with 16Gb RAM worked. Linux 32Gb get killed by the SO 🤔

Dec 02 '24 20:12 lfoppiano

On Linux, I solved by using a 32Gb RAM machine + 10Gb Swap.

Dec 03 '24 21:12 lfoppiano

Please try with our latest release gguf file https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf. thanks.

Apr 17 '25 07:04 sd983527