unsloth icon indicating copy to clipboard operation
unsloth copied to clipboard

Published 20 hours ago • verified publisher icon unslothai

Model inference - performace drop when using unsloth

Open TomekPro opened this issue 7 months ago • 4 comments

Hi, I fine-tuned a model (yam-peleg/Experiment26-7B) using unsloth. Then during inference, model correctness drops when using unsloath FastLanguageModel. I see some modules are replaced. It looks a little bit weird that for Mistral type model LlamaRotaryEmbedding is used. Any idea if this could cause a performance drop?

OLD inference: image

Unsloth way image

When comparing model files I see the following differences: image and this: image

TomekPro avatar Jul 16 '24 12:07 TomekPro