h2o-llmstudio icon indicating copy to clipboard operation
h2o-llmstudio copied to clipboard

Published 20 hours ago • verified publisher icon h2oai

[FEATURE] MoE Aux Loss

Open psinger opened this issue 1 year ago • 0 comments

🚀 Feature

Implement MoE Aux Loss for Router

https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/mixtral/modeling_mixtral.py#L76

psinger avatar Feb 06 '24 16:02 psinger