Is it possible to merge phi to llama/mistral?

[fahadh4ilyas]

I know that phi has different naming for its paramater. But, the algorithm is still transformers which contains self-attention and MLP. If we rename the parameter following llama/mistral format, is it possible to merge phi to llama/mistral?

[cg123]

Maybe at some point, but it's not hugely likely. There are two main problems that make this quite an uncertain proposition.

Firstly, it's unfortunately more than just the names that are different - Phi includes many bias terms that Llama does not, and simply dropping them would be very damaging to the model. If you went in the opposite direction and tried to rename Llama tensors to Phi conventions, you'd have a similar problem. Llama models have three matrices per MLP section, vs Phi's two, and an output projection matrix in the self-attention section that Phi also lacks. So neither architecture includes all of the operations necessary to inference both. To represent the output of the merge, you'd need to either drop important parameters (thus lobotomizing the model and necessitating extensive continued pretraining to rehabilitate it) or define a new architecture with a superset of parameters and write inference code to match.

Secondly, putting aside actual architectural differences, most merge techniques rely very heavily on the models all being fine tuned from a common ancestor. Without this we can't depend on the models being close-ish together on the loss landscape and interpolation suddenly becomes a whole lot less meaningful. There are techniques that address this, and I'm looking into them, but they need some adapting to work with transformer-based language models. This might stop being a barrier at some point, but it's a pretty big one right now.

Hope this helps, and thanks for the interest!