tensor-parallelism topic

Slicing a PyTorch Tensor Into Parallel Shards

kaiyuyue

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

bigscience-workshop

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

xrsrke

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

aniquetahir

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

InternLM