FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Published 20 hours ago • verified publisher icon lm-sys

How to use multiple Ascend NPUs?

Open litmonk opened this issue 1 year ago • 6 comments

Why is the --device npu parameter fixed to support only one Ascend NPU in code instead of multiple NPUs? def generate_stream_gate(self, params): if self.device == "npu": import torch_npu torch_npu.npu.set_device("npu:0")

litmonk avatar Feb 18 '24 02:02 litmonk

Same issue. Has this issue be solved?

QuentinWang1 avatar Apr 09 '24 08:04 QuentinWang1

I have the same issue, and hardcoding to multiple NPUs won't work either.

n1vk avatar May 17 '24 09:05 n1vk

同样的问题

zxrneu avatar Jun 07 '24 02:06 zxrneu

image image ASCEND_RT_VISIBLE_DEVICES=1

zxrneu avatar Jun 07 '24 02:06 zxrneu

image image ASCEND_RT_VISIBLE_DEVICES=1

This only changes that one NPU being used, not using multiple NPUs.

n1vk avatar Jun 13 '24 12:06 n1vk

请问问题解决了吗，我现在只能多卡加载模型，但是还是只能单卡推理

xunmenglt avatar Aug 01 '24 10:08 xunmenglt