TinyLlama LoRA Example

[fakerybakery]

Hi, I saw this example and was wondering if it might be possible to train TinyLlama with LoRA. I haven't been able to figure out how to convert the weights. Might it be possible to provide an example? Thank you! cc: @awni

[awni]

It's a great idea, we'd love to extend the LoRA example to more models. You can see how we convert TinyLlama here

[fakerybakery]

Ah, nice - before I was trying to convert it with the Phi script :). So does LoRA w/ MLX supports TinyLlama?

[awni]

There are a couple things missing in the LoRA example to work with TinyLlama (namely RoPE without traditional).

[fakerybakery]

Hi, Thank you so much! Is there a timeline for adding LoRA support for TinyLlama, now that the 3T version is released? Thank you!

[awni]

I got it working yesterday by using the conversion script in the llama example with a couple of small changes in the lora example. I think it makes sense to update the lora example to support downloading HF models directly. I can't give an exact ETA yet, but shouldn't take that long to implement.

[fakerybakery]

Thanks! Also is there a way to convert MLX -> HF?