megablocks icon indicating copy to clipboard operation
megablocks copied to clipboard

Published 20 hours ago • verified publisher icon stanford-futuredata

Can we change self.blocking in dmoe.py from 128 to 64?

Open seanM29 opened this issue 9 months ago • 2 comments

I use megablocks to implement a fine-granded moe, the ffn_hidden_size is divisible by 64, but is not divisible by 128, can we change it to 64? Thanks a lot

seanM29 avatar May 24 '24 06:05 seanM29