After finished fine-tune training.. Invalid header in file issue..!

[SOSONAGI]

Hi,

Many thanks for share this wonderful works!

I've trained mistral-7b with lora.py from the instruction and i got a adapters.npz.

I was trying to test this adapter with below commands and issues were raised.

 python lora.py --model /users/soonde17/anaconda3/envs/mlx/mlx/mlx-examples/mlx-examples/mistral-7B-v0.1 --adapter-file /users/soonde17/anaconda3/envs/mlx/mlx/mlx-examples/mlx-examples/lora/adapter_file --test

messages

Traceback (most recent call last):
  File "/Users/soonde17/anaconda3/envs/mlx/mlx/mlx-examples/mlx-examples/lora/lora.py", line 385, in <module>
    model.load_weights(args.adapter_file)
  File "/Users/soonde17/anaconda3/envs/mlx/lib/python3.10/site-packages/mlx/nn/layers/base.py", line 94, in load_weights
    self.update(tree_unflatten(list(mx.load(file).items())))
RuntimeError: [load] Invalid header in file /users/soonde17/anaconda3/envs/mlx/mlx/mlx-examples/mlx-examples/lora/adapter_file

Do i have to fix something when i fine-tune the model with lora? or retry with increase the number of iters? (I've trained 300 inters for that)

Please help to figure this out..!

Thank you.

[awni]

That is definitely not expected. The adapter file should have a npz extension (our savez automatically adds this even if you don't specify it). Could you double check that the file exists with a .npz extension?

[SOSONAGI]

Hi, I really appreciate your kind response.

I've got the .npz file as below after i've finished to train the fine-tune.

but, i got errors that i've uploaded issues two days ago.

Do i have to change something on line in lora.py or re-train that?

Thank you very much for that.