microsoft
Where can I find MCR-DL?
Hi, I need a pointer to MCR-DL. Is it open source? where can I start looking into its codebase?
@jeffra @yuxionghe
Why do you need a pointer for MCR-DL? Perhaps it would be more helpful if you could provide more details.
Hey, no specific reasons. Its mostly to ding into the code and the optimizations done by the DeepSpeed team. Is it not openly available?
Hey, no specific reasons. Its mostly to ding into the code and the optimizations done by the DeepSpeed team. Is it not openly available?
Hey @mayank31398. I'm the lead author of the MCR-DL code and paper. The DeepSpeed code for the paper is not open-source, but we will be open-sourcing a version without the mixed-backend support under my university GitHub org here: https://github.com/OSU-Nowlab
While this version doesn't (yet) allow the mixed-backend optimization, it will still allow users to decouple communication backends from PyTorch's distributed module. This enables much faster small-message performance, allows non-NCCL backends to be used with torch without messy source builds, communication logging, torch communication benchmarks, and greatly simplifies communication optimizations such as compression. I'll report back here once it's up. Expect it by the new year.
