hezq06

So does the attention head number get included?

I've tried my test script on the current master version of Deepspeed (0.7.3+aca34a9). Unfortunately, the test doesn't pass and the bug still persist.