modelmesh-serving
modelmesh-serving copied to clipboard
kserve
Need support on manipulating model to pod in order to deploy ensemble model
Describe your proposed solution
I am trying to deploy ensemble model to Triton with ModelMesh. In order to do this I need a mechanism to ensure that all the sub models have to be in a same pod. Currently, ModelMesh randomly segrerate my models into different pods so the ensemble will be failed. Do you have any plan for this in the future? If not please guide me where I can modify the source code in order to achieve this!
@vulong3896 thanks for your question. This requirement has come up before, and I'd given it quite a bit of thought some time ago (prior to model-mesh being open sourced). I've copied notes from an internal issue to a new one here: https://github.com/kserve/modelmesh-serving/issues/317.
Do you have any plan for this in the future?
No plans at this moment but if you are interested in taking it on I can help to guide/review!
@njhill My current project also needs this feature. If you have no plan for this yet, I can try implementing it. Could you give me some pointers on where to start and an estimate of the effort it would take?
