StarcoderPlus: RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[phalexo]

trafficstars

Should the same code "generate.py" be used for StarcoderPlus? Not very starry so far.

(StarCoder) developer@ai:~/starcoder/chat$ python generate.py --model_id /opt/data/data/bigcode/starcoderplus Loading dialogue template from local directory No dialogue template found in model repo. Defaulting to the no_system template. === SAMPLE PROMPT === <|system|> <|end|> <|user|> Develop a C++ program that reads a text file line by line and counts the number of occurrences of a specific word in the file.<|end|> <|assistant|>

Special tokens: {'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'additional_special_tokens': ['<|endoftext|>', '<fim_prefix>', '<fim_middle>', '<fim_suffix>', '<fim_pad>', '', '<gh_stars>', '<issue_start>', '<issue_comment>', '<issue_closed>', '<jupyter_start>', '<jupyter_text>', '<jupyter_code>', '<jupyter_output>', '<empty_output>', '<commit_before>', '<commit_msg>', '<commit_after>', '']} EOS token ID for generation: 0

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/developer/mambaforge/envs/StarCoder did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/lib/openmpi'), PosixPath('/usr/local/cuda-11.8/lib'), PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nccl/lib'), PosixPath('/home/developer/anaconda3/lib')} warn(msg) CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11.8/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 5.2 CUDA SETUP: Detected CUDA version 118 /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! warn(msg) CUDA SETUP: Loading binary /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so... Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████| 7/7 [01:52<00:00, 16.02s/it] Traceback (most recent call last): File "/home/developer/starcoder/chat/generate.py", line 139, in main() File "/home/developer/starcoder/chat/generate.py", line 119, in main generated_ids = model.generate(**batch, generation_config=generation_config) File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/transformers/generation/utils.py", line 1572, in generate return self.sample( File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/transformers/generation/utils.py", line 2655, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0