modelmesh icon indicating copy to clipboard operation
modelmesh copied to clipboard

Model per instance model-mesh by default

Open fsatka opened this issue 1 year ago • 1 comments

Now model load only on one instance, and lazy loading on another pods, when reauest has come.

Can we modify internal modelmesh parameters for default loading model on all ServingRuntime instances?

fsatka avatar Sep 25 '23 08:09 fsatka

@fsatka -- ModelMesh was designed to optimize resource utilization. Why would you want to load additional instances of the same model/predictor/ISVC on all serving runtime pods regardless of inference request traffic? Just for testing purposes?

ckadner avatar Jan 19 '24 22:01 ckadner