LongAlign icon indicating copy to clipboard operation
LongAlign copied to clipboard
verified publisher icon THUDM

Needel_test CUDA OOM 了应该怎么解决？

Open SefaZeng opened this issue 1 year ago • 1 comments

token 太多OOM了应该怎么解决？

SefaZeng avatar Jun 11 '24 09:06 SefaZeng

如果条件允许的话，可以用多gpu推理，只需要在load模型时传入device_map="auto"

bys0318 avatar Jun 12 '24 10:06 bys0318