question about training time

[harborsarah]

System Info

Dear authors,

I have a question regarding the training time utilizing the peft package. I tried using LoRA with a swin transformer to reduce the parameter size.

model = SwinModel.from_pretrained('./swin-large-patch4-window7-224-in22k').cuda()
config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
lora_model = get_peft_model(model, config)

And finally, train on the lora_model. My question is: as I tried, train 'model' and train 'lora_model' almost have the same running time, even though the parameter size is reduced from 200M to 1M. Is that normal? or did I do something wrong?

Thanks a lot for your reply.

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder
• [ ] My own task or dataset (give details below)

Reproduction

model = SwinModel.from_pretrained('./swin-large-patch4-window7-224-in22k').cuda() config = LoraConfig( r=16, lora_alpha=16, target_modules=["query", "value"], lora_dropout=0.1, bias="none", modules_to_save=["classifier"], ) lora_model = get_peft_model(model, config)

Expected behavior

Please give an explanation about this situation