MoE

Open Muennighoff opened this issue 1 year ago • 5 comments

Replaces https://github.com/allenai/OLMo/pull/541

Notes:

I didn't find norm_after to work well but added it to conform with other parts of the code but can also remove it
Only left in the config file used for the final 5T run
I didn't include all configurations that we ran for OLMoE (e.g. expert choice) - I will probably put instructions for those in a separate olmoe repository for people who want to exactly reproduce

Jun 30 '24 21:06 Muennighoff

Linking this related PR that we should merge after: https://github.com/allenai/OLMo/pull/707

If this PR here looks good to you, could you approve it @epwalsh / @dirkgr ? :)

Aug 20 '24 17:08 Muennighoff

All tests are passing except the GPU test which I assume is expected to fail. Feel free to merge 😊

Sep 04 '24 23:09 Muennighoff

What's going on with this PR? Can we merge?

Oct 03 '24 18:10 dirkgr

What's going on with this PR? Can we merge?

Fixed some basics as discussed; I think we can merge!

Oct 26 '24 04:10 Muennighoff

@dirkgr shall we merge?

Dec 13 '24 19:12 soldni