model-parallelism topic

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

microsoft

Making large AI models cheaper, faster and more accessible

hpcaitech

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

PaddlePaddle

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Oneflow-Inc

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

alibaba

A GPipe implementation in PyTorch

kakaobrain

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

ryantd

A curated list of awesome projects and papers for distributed training or inference

Shenggan

NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference

tanyuqian

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

xrsrke