world size assertionerror

[sharlec]

I try to make the 7B model on my single GPU server, and I have error:

Traceback (most recent call last):
  File "inference.py", line 82, in <module>
    run(
  File "inference.py", line 50, in run
    generator = load(
  File "inference.py", line 17, in load
    assert world_size == len(
AssertionError: Loading a checkpoint for MP=0 but world size is 1

I used the community way to download the model files.

Where can i modify the MP setting? or I have to run it with multiple GPU way?

[randyadd163]

I encountered a similar problem and was able to resolve it using this command. torchrun --nproc_per_node 1 inference.py --ckpt_dir ./pyllama_data/7B --tokenizer_path ./pyllama_data/tokenizer.model with A100(40G)*2

[sharlec]

I still have the same problem, on single 3070 8GB

[alexch33]

same problem on single 1070 8GB

solved this problem. my file with model had exttension .pt, in python code need .pth in file inferency.py in 16 line

but now i can find params.json for my model, there is only model.pth and tokenizer.model

[regstuff]

same problem on single 1070 8GB

solved this problem. my file with model had exttension .pt, in python code need .pth in file inferency.py in 16 line

but now i can find params.json for my model, there is only model.pth and tokenizer.model

Fixed the params.json by downloading the params.json here: https://huggingface.co/nyanko7/LLaMA-7B/tree/main

[MianchuWang]

Hi, I solved this issue by correcting the path ...

I ran inference.py without quantization. The command is python inference.py --ckpt_dir pyllama_data/7B --tokenizer_path pyllama_data/tokenizer.model

[sharlec]

thanks for all the comments and help here. In my scenario, it turns out my code did not detect the file properly, so the length of the files is always 0. If anyone has a similar issue, maybe check the file loading folder.