BitNet icon indicating copy to clipboard operation
BitNet copied to clipboard

Trying to setup Llama3-8B-1.58-100B-tokens with i2_s

Open yujiapingff opened this issue 3 months ago • 1 comments

I manually downloaded the model and set the model with the command "python setup_env.py -md .\models\Llama3-8B-1.58-100B-tokens -q i2_s" in Windows 11 OS. The result shows:

"ERROR:root:Error occurred while running command: Command '['./build/bin/Release/llama-quantize', '.\models\Llama3-8B-1.58-100B-tokens\ggml-model-f32.gguf', '.\models\Llama3-8B-1.58-100B-tokens\ggml-model-i2_s.gguf', 'I2_S', '1']' returned non-zero exit status 1., check details in logs\quantize_to_i2s.log"

After I check the log, it says: "main: build = 22 (bf11a49) main: built with Clang 18.1.8 for Win32 main: quantizing '.\models\Llama3-8B-1.58-100B-tokens\ggml-model-f32.gguf' to '.\models\Llama3-8B-1.58-100B-tokens\ggml-model-i2_s.gguf' as I2_S using 1 threads llama_model_quantize: failed to quantize: tensor 'output.weight' data is not within the file bounds, model is corrupted or incomplete main: failed to quantize model from '.\models\Llama3-8B-1.58-100B-tokens\ggml-model-f32.gguf'"

Am I downloading the wrong version of Llama3-8B-1.58-100B-tokens, or are other operations I made wrong?

yujiapingff avatar Nov 22 '24 16:11 yujiapingff