mlx-examples
mlx-examples copied to clipboard
TinyLlama LoRA Example
Hi, I saw this example and was wondering if it might be possible to train TinyLlama with LoRA. I haven't been able to figure out how to convert the weights. Might it be possible to provide an example? Thank you! cc: @awni
It's a great idea, we'd love to extend the LoRA example to more models. You can see how we convert TinyLlama here
Ah, nice - before I was trying to convert it with the Phi script :). So does LoRA w/ MLX supports TinyLlama?
There are a couple things missing in the LoRA example to work with TinyLlama (namely RoPE without traditional).
Hi, Thank you so much! Is there a timeline for adding LoRA support for TinyLlama, now that the 3T version is released? Thank you!
I got it working yesterday by using the conversion script in the llama
example with a couple of small changes in the lora
example. I think it makes sense to update the lora example to support downloading HF models directly. I can't give an exact ETA yet, but shouldn't take that long to implement.
Thanks! Also is there a way to convert MLX -> HF?