peft icon indicating copy to clipboard operation
peft copied to clipboard

Update OFT to fix merge bugs

Open Zeju1997 opened this issue 6 months ago • 3 comments

changes for BOFT

  • fixing small errors in comments of BOFT
  • making Conv2D operation consistent with Linear

changes for OFT

  • [Important!] fixing the incorrect merging of OFT, it is not an implementation error, but more of an incorrect understanding of how orthogonal fine-tuning works, resulting in a wrong way of merging adapters
  • update config.py/layer.py/model.py file to be consistent with other peft methods
  • adding additional paramters oft_block_size, oft_dropout, bias
  • fixing module dropout implementation error (in orthogonal finetuning, dropout is not the same as LoRA)
  • fixing errors in merge and unmerge function in OFT
  • making Conv2D operation consistent with Linear

Zeju1997 avatar Aug 08 '24 18:08 Zeju1997