accelerate icon indicating copy to clipboard operation
accelerate copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

[Feature Request] Support Megatron-Deepspeed

Open zhenlohuang opened this issue 2 years ago • 5 comments

Megatron-LM and Deepspeed is supported by accelerate in experimental state. Do you have a plan to support Megatron-Deepeed?

zhenlohuang avatar May 03 '23 07:05 zhenlohuang

cc @pacman100

sgugger avatar May 03 '23 13:05 sgugger

Hello @zhenlohuang, we want to promote and grow the usage of Megatron-LM integration at present. Things will be prioritised based on the usage and the interest of the community. This isn't in backlog at present. The community is welcome to contribute to this 🤗.

pacman100 avatar May 03 '23 17:05 pacman100

Does Megatron LM support finetuning of StarCoder? if not can you throw some light on how to do it? Thank you in advance.

KOVVURISATYANARAYANAREDDY avatar Jun 06 '23 04:06 KOVVURISATYANARAYANAREDDY

@pacman100 is it possible to use both a deepspeed plugin and megatron-lm plugin in conjunction to achieve 3-d parallelism in accelerate? Or is it possible some other way?

sam-h-bean avatar May 12 '24 04:05 sam-h-bean