arcee-ai
Examples needed.
Let's say I wish to merge LLAMA-3-8B with Mistral 7B creating a MOE.
How should I proceed?
Or for example 2 small models (3B/4B) of different architectures.
Attempt to pre-prompt your favorite Gen AI (I use Perplexity.ai) with papers on Evolutionary Model Merging. I do this by supplying the titles only of the mergekit/arcee.ai paper and the classic Evolutionary Optimization of Model Merging Recipes paper and then clearly define your small model goals. I have a small model goal of building one dimension of a "Document AI". It is comprised of merging Donut, LayoutLM and DocFormer PLUS a base model. I'm actually using Vicuna 7B. This beginning dimension of Document Ai is designed to recognize the sloppiest of document renderings so as to get text from it every single solitary time without fail. It will also interpret my self-dictation that I made when an idea emerged as I was getting a last-minute subway door B4 the train spins off. Use "in this context" and "as a follow-up" with the Gen AI a lot to assuage "catastrophic forgetting". Good luck.
@jpeek34556 was that supposed to be an answer to my question? I only understood "good luck" :P