AttributeError: 'LlamaForCausalLM' object has no attribute 'merge_and_unload'

[tshrjn]

System Info

• transformers version: 4.37.0.dev0
• Platform: Linux-5.4.0-155-generic-x86_64-with-glibc2.31
• Python version: 3.10.12
• Huggingface_hub version: 0.20.2
• Safetensors version: 0.4.1
• Accelerate version: 0.26.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

No response

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

When using the TinyLlama model with Lora training as in unsloth's colab shown in readme. On trying to 'merge_and_unload' post training, I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[43], line 1
----> 1 model = model.merge_and_unload()

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1695, in Module.__getattr__(self, name)
   1693     if name in modules:
   1694         return modules[name]
-> 1695 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'LlamaForCausalLM' object has no attribute 'merge_and_unload'

Expected behavior

To be able to merge the adaptors.

[amyeroberts]

Hi @tshrjn, thanks for raising an issue!

It looks like this is an issue on how the transformers model is wrapped on unsloth's side. Our model's don't natively have a merge_and_upload method.

[tshrjn]

@amyeroberts Thanks for taking a look.

This might be a PEFT library issue. The docs for which states the merge_and_unload function here.

[amyeroberts]

@tshrjn Thanks for the pointer! cc @younesbelkada

[younesbelkada]

Hi @tshrjn Thanks a lot for the issue! Have you correctly called a FastLanguageModel.get_peft_model before running the training? I think that unsloth properly takes care of merging lora weights in the base model if you call merge_and_unload() - Can you also print model before that error? cc @danielhanchen as well

[danielhanchen]

Oh I will check this! I'll run the notebook again to investigate :)

[danielhanchen]

I just updated the Tiny Llama notebook: https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing

Will announce tomorrow, but we now support merging to GGUF / float16 directly. The saving modules are on the very bottom.

But model.merge_and_unload() only works if calling FastLanguageModel.get_peft_model as Younes described.

I'll investigate the HF notebooks I uploaded as well + will update them with the latest Unsloth release!

[younesbelkada]

Thanks very much @danielhanchen ! @tshrjn as stated by @danielhanchen can you confirm running the cell with get_peft_model fixes your issue ? 🙏

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[itsmethahseer]

what is the version of Peft library to install for supporting merge_and_uload() method.