problem with loading my finetuned Llama2 model - type object 'Params4bit' has no attribute 'from_prequantized'

[Lujain89]

System Info

I finetuned a model

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=fp16,
    bf16=bf16,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    group_by_length=group_by_length,
    lr_scheduler_type=lr_scheduler_type,
    report_to="tensorboard",
    auto_find_batch_size=False,
)

and then pushed it HF hub:

model.push_to_hub(new_model, max_shard_size='2GB')

Then, when I tried to load it back from HF hub I encountered this error:

/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_4bit.py in create_quantized_param(self, model, param_value, param_name, target_device, state_dict, unexpected_keys) 196 unexpected_keys.remove(k) 197 --> 198 new_value = bnb.nn.Params4bit.from_prequantized( 199 data=param_value, 200 quantized_stats=quantized_stats,

AttributeError: type object 'Params4bit' has no attribute 'from_prequantized'

Reproduction

Now you can use model and tokenizer for inference

finetuned_model, f_tokenizer, peft_config = load_model("huggingfaceModelname")

Expected behavior

load the finetuned model

[liuweikarlie]

i got the same problem, did you find the solution ?

[matthewdouglas]

4bit serialization requires bitsandbytes>=0.42.0. Please verify that you have a newer version installed, or upgrade with pip install -U bitsandbytes.

[liuweikarlie]

4bit serialization requires bitsandbytes>=0.42.0. Please verify that you have a newer version installed, or upgrade with pip install -U bitsandbytes.

Thank you !