Sourab Mangrulkar

the function is `print_trainable_parameters` and not `print_trainable_parameter`

Hello @chapter544, that is a nice idea and as per what you have described it might work. You can make the embedding layers additionally trainable and add lora layers to...

Hello @chapter544, see this: https://github.com/huggingface/peft/pull/337#issuecomment-1527412343

Hello, this PR works fine for me, for debug mode you have to specify `--debug_mode`. wandb: https://wandb.ai/smangrul/huggingface/runs/3cxt4cnc?workspace=user-smangrul with bnb==0.37, the vram is stable

@ElleLeonne I did test and it does work Continuing from the ckpt of the above run via: ``` python finetune.py --base_model 'path/to/it' --data_path 'yahma/alpaca-cleaned' --output_dir './lora-alpaca-2' --debug_mode --num_epochs=3 --cutoff_len=512 --lora_target_modules="[q_proj,k_proj,...

@ElleLeonne, this gist https://gist.github.com/pacman100/8e7a6eedabf34e1a88dd74a96c3b619f should exhibit the behaviour that you are looking for. But, it doesn't make much sense to me, could you provide a concrete example of how the...

Hello, during full finetuning, the embedding layer with additional tokens is also trained which is not the case when using PEFT LoRA as per the code you shared. I think...

https://github.com/huggingface/peft/issues/349#issuecomment-1527059611

Hello @adibMosharrof, see this comment please: https://github.com/huggingface/peft/pull/337#issuecomment-1527412343