[BugFix] Illegal memory access for MoE On H20

Open Abatom opened this issue 10 months ago • 1 comments

When we attempted to deploy DeepSeek R1 671B on two 8-card H20 machines, vLLM crashed and reported illegal memory access whenever the prompt length exceeded 32K. This PR fixes the bug.

And I found that the implementation of SGLang is similar to that of vLLM, so I made the changes together.

Feb 22 '25 03:02 Abatom

cc @zhyncs

Feb 22 '25 04:02 FrankLeeeee

@Abatom Thanks for your PR! I tried the fix with https://github.com/sgl-project/sglang/issues/3333 Unfortunately the Memory access fault persists. Would you please also confirm? Thanks.

Feb 23 '25 09:02 HaiShaw

#3679 made the same fix 😂

Feb 23 '25 19:02 ch-wan

@merrymercy, Hi the same change([BugFix] Illegal memory access for MoE On H20 #13693) have already been merged in vLLM.

Mar 06 '25 07:03 Abatom

It has been merged. I added a co-author for you. Thank you.

Mar 13 '25 05:03 zhyncs

ref https://github.com/sgl-project/sglang/pull/3679

Mar 13 '25 05:03 zhyncs