建议代码更新一下

[Ambier]

用V100训练之后predict会报错，infere的代码针对V100需要添加： with torch.autocast("cuda"):

[Ambier]

在V100上完成predict代码如下： from transformers import AutoModel import torch

model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, device_map='auto') from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) from peft import PeftModel

model = PeftModel.from_pretrained(model, "./output/") import json

instructions = json.load(open("data/alpaca_data.json")) answers = [] from cover_alpaca2jsonl import format_example

query = input('Input your question:') while query != 'quit': with torch.autocast("cuda"): for idx, item in enumerate(instructions[:3]): feature = format_example(item) input_text = feature['context'] input_text = 'Instruction: %s' % query # print('InputText:%s' % input_text) ids = tokenizer.encode(input_text) input_ids = torch.LongTensor([ids]) out = model.generate( input_ids=input_ids, max_length=150, do_sample=False, temperature=0 ) out_text = tokenizer.decode(out[0]) print('OutputText:%s' % out_text) answer = out_text.replace(input_text, "").replace("\nEND", "").strip() item['infer_answer'] = answer # print(f"### {idx+1}.Answer:\n", item.get('output'), '\n\n') # answers.append({'index': idx, **item}) query = input('Input your question:')

[suc16]

如果用.half()做精度切换，和这个with autocast():是一个效果吧？ 是volta架构显卡的问题吗？

[lucasjinreal]

训练也是有问题的，v100显卡下，无法同时开启load_in_8bit 和 fp16

[zzy347964399]

训练也是有问题的，v100显卡下，无法同时开启load_in_8bit 和 fp16

你训练的时候把load_in_8bit=True改成false