[Question] 有没有低精度推理的模式呢？在哪里配置

[ZSitong]

Required prerequisites

• [X] I have read the documentation https://github.com/baichuan-inc/baichuan-7B/blob/HEAD/README.md.
• [X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [ ] Consider asking first in a Discussion.

Questions

直接按照官方代码跑起来需要28G显存，7b的模型需要这么高的显存着实不大合理，尝试用如下方法会报错： model = AutoModelForCausalLM.from_pretrained("baichuan-inc/baichuan-7B", device_map="auto", trust_remote_code=True).half()

Checklist

• [X] I have provided all relevant and necessary information above.
• [X] I have chosen a suitable title for this issue.

[sralvins]

torch_dtype=torch.float16

[ninehills]

可以参考下 https://github.com/baichuan-inc/baichuan-7B/issues/11#issuecomment-1592840751