How to see all possible lora_target_modules?

[zolastro]

I was wondering where you can find more info about the lora_target_modules. So far, I've seen that the default ones are ["q_proj",v_proj"], could you please give more insight on where to find them and how to use them?

Thanks in advance!

[zolastro]

I just found that these are references to the fields in model.named_modules(). However, I wonder if more insight in which of these modules should we train and why.

[pan365wang]

我也想知道

[ElleLeonne]

I just found that these are references to the fields in model.named_modules(). However, I wonder if more insight in which of these modules should we train and why.

Oh yes, so I discovered some more.

lm_head, and wte for attention heads and text embedding layer respectively.

Also discovered that you can just pass in model.layer too, which works so long as it's a supported type.