serve icon indicating copy to clipboard operation
serve copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

How to distribute multi models to each gpu?

Open dzcmingdi opened this issue 3 years ago • 2 comments

I have two models: model0,model1 and two gpus: gpu0,gpu1. I want to set model0 to gpu0,model0 to gpu1,it means that the work of model0 will always on gpu0 and model1 is on gpu1. How to make it? Is it possible to implement by serve configuration or handle.py? Could you help me?Thank you very much!

dzcmingdi avatar May 04 '22 16:05 dzcmingdi

Not yet unfortunately, we are planning for it https://github.com/pytorch/serve/issues/1519

msaroufim avatar May 04 '22 16:05 msaroufim

torchserve does round robin assign gpu automatically.

lxning avatar May 12 '22 01:05 lxning