megatron-lm topic

Annotations of the interesting ML papers I read

shreyansh26

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

xrsrke

A LLaMA1/LLaMA12 Megatron implement.

MoFHeka

Odysseus: Playground of LLM Sequence Parallelism

feifeibear

Super-Efficient RLHF Training of LLMs with Parameter Reallocation

openpsi-project