mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Published 20 hours ago • verified publisher icon EricLBuehler

Tensor parallel support for multi GPU

Open ilookee opened this issue 7 months ago • 10 comments

Hello, I'm not sure if multi GPU is supported yet. I didn't find parameters for tensor parallel, and the "num_device_layers" parameter seems not work. Please let me know if it supports or has plans to support multi GPU. Thanks for your awesome work!

ilookee avatar Jul 24 '24 11:07 ilookee