transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Add Qwen2MoE

Open bozheng-hit opened this issue 11 months ago • 2 comments

Adding Qwen2MoE

This PR adds the support of codes for the coming Qwen2MoE models. For information about Qwen, please visit https://github.com/QwenLM/Qwen. @ArthurZucker

bozheng-hit avatar Feb 29 '24 19:02 bozheng-hit

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

HuggingFaceDocBuilderDev avatar Mar 02 '24 04:03 HuggingFaceDocBuilderDev

Hey, I guess our update of the testing solves the issues mentioned. Take another look?

JustinLin610 avatar Mar 20 '24 08:03 JustinLin610

@ArthurZucker Hi, I have rebased our branch and solved all the conflicts. I think the codes are ready to be merged now.

bozheng-hit avatar Mar 25 '24 06:03 bozheng-hit

Thanks for your efforts! merging 🥳

ArthurZucker avatar Mar 27 '24 01:03 ArthurZucker

Hi @bozheng-hit

Thank you for adding this model 🚀

I see Qwen/Qwen1.5-MoE-A2.7B is used in the tests, but I could not find it on the Hub. See

https://huggingface.co/models?search=Qwen1.5-MoE

Could you check this, open a PR to make some necessary updates, and make sure the integration tests pass by running

RUN_SLOW=1 TF_FORCE_GPU_ALLOW_GROWTH=yes python3 -m pytest -v tests/models/qwen2_moe

Thank you in advance

ydshieh avatar Mar 27 '24 15:03 ydshieh

Didn't saw Qwen/Qwen1.5-MoE-A2.7B anywhere, time estimation on new model release?

lucasjinreal avatar Mar 28 '24 02:03 lucasjinreal