UMOE-Scaling-Unified-Multimodal-LLMs icon indicating copy to clipboard operation
UMOE-Scaling-Unified-Multimodal-LLMs copied to clipboard

Published 20 hours ago • verified publisher icon HITsz-TMG

Clarification on 3-Step Training Approach and Commands for Uni-MoE v2

Open Bhagyashreet20 opened this issue 8 months ago • 2 comments

I like the three step innovative training approach to train the MLLMs. This intrigued me more and I was going through the scripts trying to replicate 3 step training technique to train my own model. However, I have few queries.

  1. is it possible to replicate all three training steps with the scripts in uni-moe-v2 folder?
  2. Could you share the command to train uni-moe-v2-speech as there are only inference and eval scripts?
  3. relating to the 3 step training approach and the given model checkpoints, Uni-MoE 8-expert base is the result of step1, Uni_MoE 8-expert experts model after step 2 and Uni_MoE 8-expert finetune model is the model after step 3. Is my understanding correct?

Bhagyashreet20 avatar Jun 25 '24 00:06 Bhagyashreet20