mixture-of-experts topic

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

microsoft

"Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts" (NeurIPS 2020), original PyTorch implementation

mryab

Hierarchical Mixture of Experts,Mixture Density Neural Network

AmazaspShumik

Multi-Task Learning package built with tensorflow 2 (Multi-Gate Mixture of Experts, Cross-Stitch, Ucertainty Weighting)

AmazaspShumik

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

learning-at-home

A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)

drawbridge

Surrogate Modeling Toolbox

SMTorg

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

davidmrau

Tutel MoE: An Optimized Mixture-of-Experts Implementation

microsoft

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

lucidrains