main example for qLoRA fails: AttributeError: 'Model' object has no attribute 'named_parameters'

[cerisara]

Running the short example found in the main doc: https://github.com/intel/intel-extension-for-transformers/blob/main/docs/qloracpu.md

But it crashes here:

model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = True)
File "/home/xtof/nvme/envs/cpuqlora/lib/python3.9/site-packages/peft/utils/other.py", line 95, in prepare_model_for_kbit_training
for name, param in model.named_parameters():
AttributeError: 'Model' object has no attribute 'named_parameters'

Tried both with the main pip version and with today's commit 98721e6

Thank you, it's really a library many AI practitioners are waiting for!

[XinyuYe-Intel]

Hi @cerisara , thanks for your finding. Our API has changed a little bit, but we haven't been able to update the usage in the README accordingly in time, sorry for your inconvenience. You can try below code, meanwhile, we will update the README accordingly.

import torch
from intel_extension_for_transformers.transformers.modeling import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    'decapoda-research/llama-7b-hf',
    torch_dtype=torch.bfloat16,
    load_in_4bit=True,
    use_llm_runtime=False
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType
model = prepare_model_for_kbit_training(
    model, use_gradient_checkpointing=True
)
model.gradient_checkpointing_enable()
peft_config = LoraConfig(
    r=8,
    task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, peft_config)

[cerisara]

Thank you, your code is working fine up to "WeightOnlyQuant done.", but then I've got a weird error in peft/tuners/tuners_utils.py", line 341, in get_base_layer: "while hasattr(base_layer, "base_layer"):" the error is "RecursionError: maximum recursion depth exceeded while calling a Python object"

which is true, as I also see "weight = base_layer.weight [Previous line repeated 975 more times]"

Note that I'm loading the HF Llama-2-7b-hf model, not the one you use in your example.

[XinyuYe-Intel]

Thank you, your code is working fine up to "WeightOnlyQuant done.", but then I've got a weird error in peft/tuners/tuners_utils.py", line 341, in get_base_layer: "while hasattr(base_layer, "base_layer"):" the error is "RecursionError: maximum recursion depth exceeded while calling a Python object"

which is true, as I also see "weight = base_layer.weight [Previous line repeated 975 more times]"

Note that I'm loading the HF Llama-2-7b-hf model, not the one you use in your example.

Hi, I tried Llama-2-7b-hf with above code also, haven't met the issue you mentioned, maybe you can try update peft, the peft I used is v0.6.2.

[cerisara]

Thank you, my peft was more recent: 0.8.2 I tried to downgrade peft to 0.6.2, and it works!

So you're right, it's indeed a problem of versions... But I guess making the Intel lib compatible with the most peft would be better in the long term (or independent of it ;-) )

Thank you!