Can't run LLAMA model

[helikopterodaktyl]

Describe the bug

I am getting this error when trying to run LLAMA. I can run the default models available from download_models.py like OPT or Pythia with no problem.

Traceback (most recent call last): File “/home/haxx/text-generation-webui/server.py”, line 85, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name) File “/home/haxx/text-generation-webui/modules/models.py”, line 56, in load_model model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{shared.model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=trust_remote_code) File “/home/haxx/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 471, in from_pretrained return model_class.from_pretrained( File “/home/haxx/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/modeling_utils.py”, line 2405, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models/Neko-Institute-of-Science_LLaMA-7B-4bit-32g.

I was following these instructions: https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model

tried to download this model, and indeed it doesn't have those files https://huggingface.co/Neko-Institute-of-Science/LLaMA-7B-4bit-32g . How to run this model?

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Use download_models.py to download model https://huggingface.co/Neko-Institute-of-Science/LLaMA-7B-4bit-32g Try to run the model in WebUI

Screenshot

No response

Logs

Traceback (most recent call last):
File “/home/haxx/text-generation-webui/server.py”, line 85, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name)
File “/home/haxx/text-generation-webui/modules/models.py”, line 56, in load_model
model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{shared.model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=trust_remote_code)
File “/home/haxx/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 471, in from_pretrained
return model_class.from_pretrained(
File “/home/haxx/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/modeling_utils.py”, line 2405, in from_pretrained
raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models/Neko-Institute-of-Science_LLaMA-7B-4bit-32g

System Info

Linux Mint 21.1
AMD Radeon 6800XT

[Ph0rk0z]

Do you have a 4bit version and it's auto loading a FP16? Make sure you set wbits in the UI or on the command prompt. I have this problem too.. in the UI I set the right stuff and click save then reload.

[Deca]

Same error on fresh oobabooga install. Already checked to have the right parameters in config-user.yaml

[Ph0rk0z]

Check wbits is set correctly and reload. No other way.

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.