Add Qwen support

[chenht2026]

It works. Qwen's tokenizer is based on tiktoken, I add the tokenizer(tokenization_qwen.py) from its huggingface repo without any revision. This make the code a little complicate, so maybe do not merge.

May be some one needs it.

Closes #840

[lantiga]

Thank you @chenht2026, sorry for the wait. A few of us took a break :-)

[samuelazran]

It works. Qwen's tokenizer is based on tiktoken, I add the tokenizer(tokenization_qwen.py) from its huggingface repo without any revision. This make the code a little complicate, so maybe do not merge.

May be some one needs it.

Closes #840

Would it work with Qwen 2 (Qwen/Qwen1.5-7B-Chat)? if not what needs to be added? Qwen1.5 improve a lot on its predecessor in performance. https://huggingface.co/Qwen/Qwen1.5-7B-Chat/tree/main

[rasbt]

Hey just pinging to see if you are still interested in pursuing this PR. Personally, I think it'd be awesome to support the Qwen models (1.5 and especially 2) in LitGPT. There have been some improvements in the tokenizer in LitGPT recently that could now make this more easily possible.

Btw if rebasing here based on the main branch (which changed a lot) is too messy, you could also just open a fresh PR.