Daniel Han

Oh yep saw it! We might instead be adding LoRA+, which has similar results. Technically DoRA is already enabled in Unsloth, but just not optimized

@sorasoras yes but it's not that optimized - only somewhat

@adamo1139 It should work QLoRA + DoRA - but not optimized - simply turn it with `use_dora = True` Unsure on that exact error msg, but if you use our...

@songkq Oh it should be supported if you use Llama-Factory's llamafy script. Ie maybe try https://huggingface.co/models?search=qwen%20llama. On the other hand, if some don't exist, you can try out Llama-Factory's script...

@songkq Oh wait actually on further investigation Qwen 1.5 / 2 has combined sliding window and normal attention right - I think on shorter context windows it works, but larger...

@NilanEkanayake So there are some pre converted Qwen models on HuggingFace if you search for "qwen llama". In terms of Qwen 1.5 / 2 - if it's a top feature...

@Minami-su Oh interesting llama-fying it made it worse? That's very unexpected

I'll see what I can do for Qwen 1.5/2 :))

@whyiug Hmmm I'm expecting most people to move over to Llama-3 - unless if Qwen is still needed by the community, I can work on it, but generally Llama-3 is...

Can take another look at Qwen!