Does Clipper support multiple workers?

[aksnzhy]

I have 4 GPUs on my PC and I want to use them to serve one model in a load-balanced way. Does Clipper support that?

[dcrankshaw]

Yes it does. You can create multiple replicas of a model and Clipper will automatically load-balance requests between them. However, Clipper does not yet support deploying models on GPUs automatically. You can instead do this manually by starting the Docker containers yourself using nvidia-docker after registering your model. You can look at the function that Clipper calls to start containers to see which labels and environment variables are normally set.

[lgendrot]

I may be wrong, but instead of starting the docker containers yourself manually, you can simply make the nvidia-docker runtime the default docker runtime

I haven't tested it, but presumably this would make every docker container started on your system have GPU access.