Trained model output seems illegible

[KKcorps]

I trained both redpajama-3b and llama-7b for about 1000 iterations.

The loss converged to a pretty low value (0.3) by this time.

But when I load the adapter and try to predict something (using inference notebook code as well as generate.py), I get random unicode tokens in my ouput. Then english answers are not related too much to the original code.

What can be the reason for this?

[phalexo]

7B is a tiny model. You cannot expect any quality in the output.

[KKcorps]

No I meant it should still be slightly relatable and not filled with gibberish tokens.

[amdnsr]

@KKcorps were you able to find the issue and get any meaningful output from the trained models?

[Birch-san]

@KKcorps this happened when my learning rate was too high.

[KKcorps]

@KKcorps this happened when my learning rate was too high.

I was able to solve this using a different base model. but if learning rate it too high then the training/validation loss should also not keep on dropping right?