unsloth
unsloth copied to clipboard
unslothai
[REQUEST] Support for Yarn context extension method
I would like to request the support for Yarn, would be nice to fine tune models such as https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k using unsloth.
I am even willing to help with the implementation and testing
@thedarkzeno Sorry on the delay!! Interesting on Yarn! I think I saw this metholodgy floating around, but I've never gotten round to them - I'm not an expert in Yarn, so it'll be super cool if you can give me a run down onw how Yarn works - I know Kaiokendev's RoPE Scaling works by dividing by a larger scale, ie [0, 1, 2, 3] becomes [0, 0.5, 1, 1.5, 2, 2.5, 3]. Unsure on Yarn - is this like some dynamic method like NTK?
YaRN improves upon this with "NTK-by-parts" interpolation, which selectively scales dimensions based on their frequency. By looking at the unsloth code I believe all we need is to set "trust_remote_code=True" when we load the model.
@thedarkzeno Hey! Do you know if it works right now? I tried to set the flag but still experience issues, even during inference
