DeeperSpeed
DeeperSpeed copied to clipboard
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
DeeperSpeed
DeeperSpeed is a fork of Microsoft's Deepspeed library that is tailor-made for the GPT-NeoX by EleutherAI.
Prior to 3/9/2023, DeeperSpeed was based on an old version of DeepSpeed (0.3.15). In order to migrate to the latest upstream DeepSpeed version while allowing users to access the old versions of GPT-NeoX and DeeperSpeed, we have introduced two versioned releases for both libraries:
- Version 1.0 of GPT-NeoX and DeeperSpeed maintain snapshots of the old stable versions that GPT-NeoX-20B and the Pythia Suite were trained on.
- Version 2.0 of GPT-NeoX and DeeperSpeed are the latest versions built on the latest DeepSpeed, and will be maintained going forward.