Additional parameters to mlx_lm lora? r, lora_alpha, lora_dropout, scale?

[ivanfioravanti]

When playing with fine-tuning sometimes I change from_linear in lora.py to play with them. Should we add command line args for these?

[awni]

Yes that's been on our list to add for a while.

Though, do you use mlx_lm.lora or just the original lora.py script? My preference is to update the package since that one has many more features.

[ivanfioravanti]

I always us mlx_lm now, it's becoming more powerful at each release 💪

[ivanfioravanti]

It would be great adding the new LR Scheduler or even the optimizer, but parameters become too complex then. Probably something like PR #235 to read from a config file would be better.

[awni]

Agree I think we may need to start using a yaml config

[chimezie]

#235 is dated. I can rebase it to mlx-examples/main and update it (to support the parameters that have been added since I last worked on that PR) if there is interest. I have found it useful to pull parameters from yaml and override them from what is provided via the command line.

[Solido]

I too migrated to mlx_lm.lora and end up using a shell script generator. #235 would simplify the whole process as use cases for MLX grow and simplify the tuning by having different models/confs/benchmark in parallel.

[ivanfioravanti]

@awni I think we can close this issue, in recent versions (thx @chimezie) -c parameter has been added to mlx_lm.generate and I added dropout in this pr #599

[awni]

Yes indeed.. safe to close, thank you!