Is it supported to merge multiple models into a Mixtral model?

[win10ogod]

Is it supported to merge multiple models into a Mixtral model?

[cg123]

It is now! There's a script mergekit-moe on the branch mixtral now. Keep in mind it's still early and might have problems. There is a little documentation here.

[fakerybakery]

Thank you so much! Might it be possible to merge this into the main branch?

[fakerybakery]

Hi @cg123, also, would you mind providing an example for the YAML config file for a MoE model? Also, an example of the positive prompt? Thanks!

[ddh0]

@fakerybakery here is an example config.yml from Undi: https://huggingface.co/Undi95/Mixtral-8x7B-MoE-RP-Story/blob/main/config.yaml