minLoRA

minLoRA copied to clipboard

Doesn't work with DataParallel

4

Minimum example ```python import torch import timm from torch import nn from minlora import add_lora, get_lora_params, get_lora_state_dict model_timm = timm.create_model("vit_large_patch14_clip_336.openai", pretrained=True, num_classes=0, global_pool='avg') add_lora(model_timm) model_timm = nn.DataParallel(model_timm, device_ids=[0,1]).cuda() with torch.no_grad():...

i7p9h9

amichay-feldman

Understanding the forward operation

1

I noticed that you apply the mul operation in LoraA and LoraB, then, you sum the result with the input.  I think the result of multiplying LoraA and LoraB...

evertonaleixo

您好，请问您对blip2模型的加速研究有没有什么进展和思路，是否可以交流一下呢，万分感谢

liangwanping

Freeze manually

4

Hi, thank you for your great work. I want to use yours for my experiment. I wonder get_lora_params() would load parameters to optimizer, but if the model itself can compute...

G-JWLee

Can this be used with FSDP? I haven't seen any examples of using `torch.nn.utils.parametrize` with FSDP.

ariesadel

How to use this with nanoGPT

1

Hey there, This looks like a cool project! How do we use this for nanoGPT? :)

vgoklani