quanto
quanto copied to clipboard
huggingface
qint4 failed for diffusers: QBitsTensor cannot be changed
When I used qfloat8 to quantize the unet model of Kolors-diffusers, it works well. But failed with qint4.
use qint4/(qfloat8)
class KolorsUNet2DConditionModel(QuantizedDiffusersModel): base_class = UNet2DConditionModel model = UNet2DConditionModel.from_pretrained("./Kolors-diffusers", variant="fp16", subfolder="unet") qmodel = KolorsUNet2DConditionModel.quantize(model, weights=qint4) # qfloat8 qmodel.save_pretrained("./kolors-qint4") # kolors-qfloat8
work well in qfloat8 but failed with qint4
unet = KolorsUNet2DConditionModel.from_pretrained('./model_zoo/kolors-int4') #unet = KolorsUNet2DConditionModel.from_pretrained('./model_zoo/kolors-qfloat8') pipe = KolorsPipeline.from_pretrained( KOLORS_MODEL_PATH, unet=unet._wrapped.to(dtype=torch.float16), text_encoder=text_encoder, torch_dtype=torch.float16, # torch.float16 variant="fp16",# use_safetensors=True ).to('cuda')
error log
Traceback (most recent call last):
File "/mnt/data/liyihao/workspace/text2img/text2img_kolors_v5_light.py", line 101, in
