notebooks icon indicating copy to clipboard operation
notebooks copied to clipboard

Fix Mistral v0.3 chat template and tokenizer issues

Open Imagineer99 opened this issue 5 months ago • 1 comments

Issue: Weird repetitive outputs when using the Mistral 7B v0.3 conversational notebook before and after training when using chatml template

The tokenizer was using control token 770 for padding instead of the EOS token, which was messing up training. Also the ShareGPT format wasn't converting properly to what Mistral v0.3 expects.

What I changed: Made the pad token use EOS instead of the control token Added manual conversion from ShareGPT to Mistral format as a workaround
Lowered the learning rate so gradients don't explode

The manual format conversion is kinda hacky but works until we figure out why the automatic conversion broke. Probably some update for newer models messed with the older ones.

Other conversational and mistral v0.3 notebooks need to checked for similar issues.

Imagineer99 avatar Jul 20 '25 22:07 Imagineer99

Hello, thank you for the PR

I just checked and after finetuning, the current notebook is working fine and able to stop.

Also you did not really want to use pad_token=eos_token or that would make the model did not know when to stop, since pad_token is ignored during training/fine tuning.

And yeah, Mistral chat template is a bit clunky. I don't think it's designed for multi turn conversational? Therefore we use chatml in this example. Altho, probably it's better to start from base model instead of instruct.

Erland366 avatar Aug 28 '25 16:08 Erland366