mergekit icon indicating copy to clipboard operation
mergekit copied to clipboard

Published 20 hours ago • verified publisher icon arcee-ai

Evolutionary Merging out of memory

Open ArcherShirou opened this issue 8 months ago • 4 comments

When attempting to merge three 14B LLMs on a custom task using the mergekit-evolve method, I ran into memory overflow issues on 8 A100 GPUs, each with 80G of memory. Could you please let me know if 4-bit quantized model merging is supported, or if there are any effective solutions to address this memory overflow issue?

ArcherShirou avatar Jun 04 '24 03:06 ArcherShirou