RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[LovingThresh]

RuntimeError: probability tensor contains either inf, nan or element < Whatever I input, it will raise this RuntimeError Human: what can you do? Assistant: │ 101 │ token = int(torch.argmax(last_token_logits))
│ 102 │ else: │ 103 │ probs = torch.softmax(last_token_logits / temperature, dim=-1) │ 104 │ token = int(torch.multinomial(probs, num_samples=1)) │ 105 │ │ 106 │ output_ids.append(token)

╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: probability tensor contains either inf, nan or element < 0

[zhisbug]

what hardware are you running on?

[LovingThresh]

i try it in a NGC docker in Ubuntu 22.04. I also try Windows，but all fail.

| | 刘烨 中南大学本科生 | | 15200945180 @.*** |

---- Replied Message ---- | From | Hao @.> | | Date | 04/08/2023 05:05 | | To | @.> | | Cc | LIU @.>@.> | | Subject | Re: [lm-sys/FastChat] RuntimeError: probability tensor contains either inf, nan or element < 0 (Issue #272) |

what hardware are you running on?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[djaffer]

https://github.com/lm-sys/FastChat/issues/153

[oasis-0927]

#153

@djaffer I followed the added_tokens.json way and sadly it didn't work on my case. Did this work on yours?

[djaffer]

no it didn't.

[oasis-0927]

Found a solution that might work. I downloaded the vicuna weights from https://huggingface.co/AlekseyKorshuk/vicuna-7b/ instead of applying the delta weights on llama-7b model and it worked. My guess is that this error is result from either the apply_delta process or the llama-7b model is not compatible with the vicuna-delta weights.

Hope it helps.

[LovingThresh]

thank you！but we use the 13B, and where can find the weight.

| | 刘烨 中南大学本科生 | | 15200945180 @.*** |

---- Replied Message ---- | From | @.> | | Date | 04/11/2023 17:23 | | To | @.> | | Cc | LIU @.>@.> | | Subject | Re: [lm-sys/FastChat] RuntimeError: probability tensor contains either inf, nan or element < 0 (Issue #272) |

Found a solution that might work. I downloaded the vicuna weights from https://huggingface.co/AlekseyKorshuk/vicuna-7b/ instead of applying the delta weights on llama-7b model and it worked. My guess is that this error is result from either the apply_delta process or the llama-7b model is not compatible with the vicuna-delta weights.

Hope it helps.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[oasis-0927]

@LovingThresh Maybe this solution is worth a try. https://zhuanlan.zhihu.com/p/620801429

[LovingThresh]

Thank you @JuntingGuo, It work in CPU; single GPU int8, but not in muli GPUs. It looks some works to do?

[zhisbug]

Is this issue resolved for all of you? Have you tried the new weights and apply_delta script since our v1.1 release?

[LovingThresh]

this problem has been solved.

---- Replied Message ---- | From | Hao @.> | | Date | 04/21/2023 09:58 | | To | @.> | | Cc | LIU @.>@.> | | Subject | Re: [lm-sys/FastChat] RuntimeError: probability tensor contains either inf, nan or element < 0 (Issue #272) |

Is this issue resolved for all of you? Have you tried the new weights and apply_delta script since our v1.1 release?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[zhisbug]

Great to hear @LovingThresh !

[ZhaoChuyang]

Meet the same problem, solved by load model in single GPU in 8bit. 4bit will cause the problem.