Alexander Zhipa

@tjruwase Here's the new PR so the same patch as before, sorry again.

Please let me know if more work is required for this change to get merge, @tjruwase .

Hi @tjruwase, looks like seemingly unrelated `TestHybridEngineTextGen` test keeps failing. Is this the reason why this change cannot be merged?

@tjruwase there's some weird CUDA OOM going on, where a test (`TestCompression.test_conv1d_convertion`) I didn't even affect with my changes fails. Is the testing instance being shared by multiple CI workflows?

Reducing micro batch size to try and avoid OOM. Please let me know if there's any concern with this, @tjruwase Just trying to get this fix merged.

Still no luck, @tjruwase . Is there a known issue with CUDA OOM in DeepSpeed tests? I only added a few layers to SimpleMoEModel that is not that widely used,...

Ok, looks like the checks run fine this time, @tjruwase

Hi @cli99, can you please have a look and let me know what you think?

Sorry to bug, @cli99, but is the list of reviewers for this file accurate or should I reach out to someone else? Thanks!

Sure @cli99, are you in favor of the proposed changes though? Now I also need to find a way to make it reflect the expert parameters.