Questions about original_module and modules_to_save.default

[dengchengxifrank]

System Info

• transformers version: 4.36.2
• Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.9.19
• Huggingface_hub version: 0.24.6
• Safetensors version: 0.4.5
• Accelerate version: 0.21.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?: \

Who can help?

@BenjaminBossan @sayakpaul

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder
• [X] My own task or dataset (give details below)

Reproduction

I added a module to the original network of the whisper model, and added it to modules_to_save in the process of using Lora, but when I use

 for name, params in model.named_parameters():
        print('name ',name)

I found that the model will output base_model.model.model.encoder.encoder_prompt_sid.original_module.weight and model.model.encoder.encoder_prompt_sid.modules_to_save.default.weight. I want to know the relationship between these two weights and why these two modules appear. My code is as follows:

pattern1 = 'encoder_prompt_sid'  
modules_to_save = [pattern1]
config = LoraConfig(r=8, lora_alpha=16,target_modules=["q_proj", "v_proj","k_proj","out_proj"],modules_to_save=modules_to_save,lora_dropout=0.05, bias="none")
model = get_peft_model(model, config)

Expected behavior

I want to know the relationship between modules_to_save.default.weight and original_module.weight, will they always remain the same? Or will only modules_to_save.default.weight be updated? And does original_module.weight still exist in the model?

[BenjaminBossan]

original_module.weight is a proxy for the currently active adapter. So in general, it will point to modules_to_save.default.weight but if you activate a different adapter or deactivate the adapter, that can change.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.