Baichuan2 icon indicating copy to clipboard operation
Baichuan2 copied to clipboard

Published 20 hours ago • verified publisher icon baichuan-inc

baichuan-7B-chat-4bits跑不通

Open Yining0907 opened this issue 1 year ago • 4 comments

help，根据issues里下了bitsandbytes==0.41.1，但是还是报错Baichuan7BChatTextGeneration: 'NoneType' object is not subscriptable

Yining0907 avatar Jan 26 '24 10:01 Yining0907

好像pytorch_model里面存储的quant_state的类型错误了，应该是QuantState,但是从模型中加载出来的是一个List

sxndqc avatar Jan 27 '24 01:01 sxndqc

好像pytorch_model里面存储的quant_state的类型错误了，应该是QuantState,但是从模型中加载出来的是一个List

所以应该怎么修改呢？

Yining0907 avatar Jan 30 '24 03:01 Yining0907

@Yining0907 bitsandbytes 0.41.0是支持的，但是accelerate0.26.2最近一次更新出了点bug，我提了还没pull进去

sxndqc avatar Feb 09 '24 16:02 sxndqc

我的报这个错有大佬看看吗？ValueError: We need an offload_dir to dispatch this model according to this device_map, the following submodules need to be offloaded: model.layers.2, model.layers.3,..., model.norm, lm_head.

qingchen177 avatar Feb 27 '24 07:02 qingchen177