Fulltune training Mistral 7B using JSON dataset instead of JSONL makes the model incoherent

[l3utterfly]

Please check that this issue hasn't been reported before.

• [X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Training Mistral 7B with example settings should work.

Current behaviour

I'm using my own data in ShareGPT format, training using the default config in this repo (sample packing true), and deepspeed zero3 config.

I see loss decrease steadily from 10 -> 6 during training. I saved a checkpoint and put the full-tuned model into oogabooga GUI to test it. The output becomes incoherent. It outputs "a", "of", "and", then stops.

I would at least expect it to be similar to the original model.

What could be the issue? Is it a training issue or an inference issue on the GUI side?

Steps to reproduce

see above

Config yaml

No response

Possible solution

No response

Which Operating Systems are you using?

• [X] Linux
• [ ] macOS
• [ ] Windows

Python Version

3.10

axolotl branch-commit

main

Acknowledgements

• [X] My issue title is concise, descriptive, and in title casing.
• [X] I have searched the existing issues to make sure this bug has not been reported yet.
• [X] I am using the latest version of axolotl.
• [X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

[NanoCode012]

Could you check your prompt format is the same? Your loss also starts quite high.

[l3utterfly]

My prompt is in the normal shareGPT format. I'm not using JSONL though, perhaps that could be an issue. I will try using JSONL and report back

[mxbi]

@l3utterfly Did you ever find out anything more about your issue?

Out of curiosity, what happens if you fine-tune for a small number of epochs with learning rate 0? If the model is still incoherent, then this would point to an issue somewhere with axolotl rather than your data.

[l3utterfly]

@mxbi I figured out the issue: I re-formatted my dataset to JSONL and the training works. So it seems JSON datasets will not work, only JSONL.

[ehartford]

Oh yeah I had the same thing

[NanoCode012]

Did not know that using JSON would cause such an issue. That sounds weird.

I will close this issue. Please re-open if the problem re-occurs.

[mxbi]

Surely this should be re-opened (and likely renamed): there is a real bug here in that JSON - suggested by the documentation - causes silent failure. I suspect many people are hitting the same issue and wasting time or giving up.

Do we have any example ymls where jsonl works but json does not?

[NanoCode012]

Sorry for that. The issue is reopened. Could you please provide an example config of where json does not work?

The dataset handler for json and jsonl is the same via the upstream datasets library.

[NanoCode012]

@l3utterfly , do you perhaps still have the offending dataset to share or some sample of it for reproducing this?