Zhisheng
Zhisheng
难道v100对int8的模型不适配？有没有说具体原因，能给个链接？
chatglm-6b, not quantized version. Downloaded from huggiingface [https://huggingface.co/THUDM/chatglm-6b](url). And this bug occur occasionally, for example: give an input "撰写一篇关于最近去夏威夷旅行的引人入胜的旅游博客文章，重点介绍文化体验和必看景点。" the first time I run model.chat(tokenizer, input_text, num_beams=2, history=[]), it is...
well, I run with A100, the same problem occured
according to here: https://huggingface.co/docs/transformers/main_classes/text_generation beam-search multinomial sampling by calling [beam_sample()](https://huggingface.co/docs/transformers/v4.28.1/en/main_classes/text_generation#transformers.GenerationMixin.beam_sample) if num_beams>1 and do_sample=True it seems use multinomial sampling will give "RuntimeError: probability tensor contains either `inf`, `nan` or element...
看起来还是13B效果稍微好一点，然后调大beam_num 也有帮助。
我重新在A100进行了测试 do_sample=False, num_beams=2, 没有问题 do_sample=True, num_beams=1, 没有问题 do_sample=True, num_beams=2, 有问题 根据 https://huggingface.co/docs/transformers/main_classes/text_generation 就是说进行 multinomial sampling 就会报错。目前不知道怎么解决
same problem