data-parallelism topic

Multi-GPU training for Keras

kuixu

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

ryantd

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

xrsrke

Distributed training (multi-node) of a Transformer model

hkproj

SC23 Deep Learning at Scale Tutorial Material

NERSC