mlx-examples
mlx-examples copied to clipboard
Allow the entire model to be targed for LoRA and DoRA fine tuning: LoRA and DoRA embeddings with small DoRALinear bug fix
- Added in
LoRAEmbeddingandDoRAEmbeddingwith tests so that embeddings can be targeted for fine tuning. - Added in the ability to target all
LinearandEmbeddingmodules regardless of if they are inmodel.layersallowing both the embeddings and thelm_headto be targeted for fine tuning, allowing a nearly full LoRA or DoRA fine tune of the model. - Fixed a bug with
DoRALinearthat sets the wrongself.mvalue due to it not being recalculated when theLinearlayer is changed inDoRALinear.from_linear
I checked huggingface's PEFT library for how they handle DoRA for embeddings and there is still an open ticket for it. I wasn't able to find any reference implementations, so this could be the first example of that.