mlc-llm icon indicating copy to clipboard operation
mlc-llm copied to clipboard

Published 20 hours ago • verified publisher icon mlc-ai

Dose mlc-llm support parallelism like multi-gpu, multi-node ?

Open xiongjun19 opened this issue 2 years ago • 3 comments

Dose mlc-llm support parallelism like multi-gpu, multi-node ?

xiongjun19 avatar May 05 '23 03:05 xiongjun19

Not yet, distributed inference is indeed an interesting topic that is not supported in TVM yet.

yzh119 avatar May 05 '23 05:05 yzh119

Not yet, distributed inference is indeed an interesting topic that is not supported in TVM yet.

Ok thanks

xiongjun19 avatar May 05 '23 06:05 xiongjun19

At this moment, this project focuses on single consumer-class GPU, making it possible for everyone to run on their own laptops and phones. We will bring in distributed inference later

junrushao avatar May 08 '23 22:05 junrushao

This is now supported

tqchen avatar Oct 24 '23 13:10 tqchen