ml-explore
lora example fails with ValueError when using the latest mlx release
This recent commit to mlx adds a strict mode (true by default) that checks to ensure the weights passed exactly match the parameters of the model. This causes the following error when running the lora example:
Traceback (most recent call last): File ".../mlx-examples/lora/lora.py", line 390, in <module> model.load_weights(args.adapter_file) File ".../mlx-examples/mlxenv/lib/python3.11/site-packages/mlx/nn/layers/base.py", line 149, in load_weights raise ValueError(f"Missing parameters: {missing}.") ValueError: Missing parameters: layers.24.attention.wk.weight layers.16.attention.wq.linear.weight ...
In the case of LoRA, we expect there to be missing parameters. Setting strict=False when calling model.load_weights fixes the issue. I can submit a PR to fix this if it's not already in the works.
Oh good, catch! I will add it to an outstanding diff (#219) I have to update that example. Thank you!
Thanks for this issue, glad to see a fix on the way! I just spent a lot of time today figuring this one out. :D
This is a great project, happy to help!
@awni I see in your commit that you fixed model.load_weights on line 397. While you're at it, you should fix line 379 as well so that resuming training will work.