Not find the models...

[luussta]

Describe the bug

Y use the commands: cd text-generation-webui call python server.py --chat --pre_layer 31 --wbits 4 --groupsize 128 --model gpt4-x-alpaca-13b-native-4bit-128g

And the error name: Could not find the quantized model in .pt or .safetensors format, exiting...

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

cd text-generation-webui call python server.py --chat --pre_layer 31 --wbits 4 --groupsize 128 --model gpt4-x-alpaca-13b-native-4bit-128g

Screenshot

Could not find the quantized model in .pt or .safetensors format, exiting...

Logs

Could not find the quantized model in .pt or .safetensors format, exiting...

System Info

Gpu: Nvidia 3050
Ram: 8gb

[myluki2000]

How does your models folder look like?

[luussta]

[Juqowel]

cd text-generation-webui call python server.py --chat --pre_layer 31 --wbits 4 --groupsize 128 --model gpt4-x-alpaca-13b-native-4bit-128g

Model name must match folder name:

--model anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g

[luussta]

Putting that command, it finds the model but says: DefaultCPUAllocator, any idea for how to fix it?

[Juqowel]

4-bit models get loaded into RAM before being sent to VRAM. 16GB RAM - minimum for 13b-4bit model. (10-11GB free RAM).

[luussta]

What can i do?

[luussta]

Anyone has a solution?

[myluki2000]

get a better computer or run the AI using Google Colab.

[luussta]

get a better computer or run the AI using Google Colab.

I will use google colab. 😢

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.