mlx-examples
mlx-examples copied to clipboard
Additional parameters to mlx_lm lora? r, lora_alpha, lora_dropout, scale?
When playing with fine-tuning sometimes I change from_linear in lora.py to play with them. Should we add command line args for these?
Yes that's been on our list to add for a while.
Though, do you use mlx_lm.lora
or just the original lora.py
script? My preference is to update the package since that one has many more features.
I always us mlx_lm now, it's becoming more powerful at each release 💪
It would be great adding the new LR Scheduler or even the optimizer, but parameters become too complex then. Probably something like PR #235 to read from a config file would be better.
Agree I think we may need to start using a yaml config
#235 is dated. I can rebase it to mlx-examples/main and update it (to support the parameters that have been added since I last worked on that PR) if there is interest. I have found it useful to pull parameters from yaml and override them from what is provided via the command line.
I too migrated to mlx_lm.lora and end up using a shell script generator. #235 would simplify the whole process as use cases for MLX grow and simplify the tuning by having different models/confs/benchmark in parallel.
@awni I think we can close this issue, in recent versions (thx @chimezie) -c parameter has been added to mlx_lm.generate and I added dropout in this pr #599
Yes indeed.. safe to close, thank you!