DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Illegal memory access error when infering input of length 100K

Open frankxyy opened this issue 8 months ago • 4 comments

Hi，I served this model from huggingface: 01-ai/Yi-6B-200K.

When requesting for input of length 100K，this error occurs: 2e3b5e25a7c6f9b75d964803b0313903

frankxyy avatar Nov 15 '23 11:11 frankxyy

Hi @frankxyy we do not currently support this model type. You should see an error when you try to load the model with MII: ValueError: Unsupported model type Yi

mrwyattii avatar Nov 16 '23 20:11 mrwyattii

@mrwyattii Sorry, I posted the wrong name of this model in the first post... The actual name is larryvrh/Yi-6B-200K-Llamafied. This model is actually a llama model.

frankxyy avatar Nov 17 '23 01:11 frankxyy

Got it, I will update and try again.

In the meantime, can you confirm what kind of hardware you are running on. I think from what I saw in another issue, you are running on 4xA10. Is that correct? Also, can you confirm that you are setting the max_length to larger than 100K when you run the prompt? Thanks

mrwyattii avatar Nov 21 '23 19:11 mrwyattii

@mrwyattii hi，yes I am running on 4xA10. And I will try to set larger max_length argument to test again when i am free. Thank you a lot

frankxyy avatar Nov 22 '23 03:11 frankxyy