llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

dmoe from hf

Open pretidav opened this issue 7 months ago • 1 comments

It is not clear if it is possible to convert a MOE model from hf, say Mixtral 8x7B into dmoe layers and exploit expert parallelism. Is it possible to start from such architecture, or the model must be defined from scratch (as in "testing_moe.yaml" ? thank you.

pretidav avatar Mar 14 '25 08:03 pretidav