Using LoRA with CodeGen 2B mono

[anujsahani01]

Fine-tuning Salesforce/codegen-2B-mono on my own text to code Python dataset. I quantized the model using HuggingFace transformer's BitsAndBytesConfig and LoRA config.

Implemented loRA config by using the following code:

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=[],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

I am not sure what should i be using in target models, in official git provided below.

Target modules for CodeGen is not mentioned.

peft git repo

Please help me out with this. Any of your inputs will be highly appreciated. Thank You!

[peppa-xyz]

same issue

[anujsahani01]

same issue

Using target_modules= ["q_proj", "v_proj"], as target module worked for me but this is specified for bart model and not for CodeGen.

You may have a look here: https://github.com/huggingface/peft/blob/main/src/peft/utils/other.py#L262