Multi-GPU support for larger-than-VRAM models

[emvw7yf]

Awesome project, thanks!

Does it support sharding large models across multiple GPUs, or would this be in scope for this project in the future?

[tqchen]

thank you for your suggestion, yes we love to support the needs of the community and bring it in our roadmap

[earonesty]

llama.cpp seems to support this now fyi!

[tqchen]

Multi gpu now lands in mlc