server
server copied to clipboard
triton-inference-server
Allow triton to read from multiple cloud model repositories
Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] Trying to use Triton with ClearML-serving. Currently, there are means (with some coding) to copy the models from ClearML to Triton. However, I am looking for a means to differentiate manually deployed models vs those from ClearML. E.g. From two S3 buckets or a hybrid of local folder + S3 bucket.
Describe the solution you'd like A clear and concise description of what you want to happen. Allow Triton to take in multiple repos.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. Nil.
Additional context Add any other context or screenshots about the feature request here.
Triton already supports such feature which can be done by specifying --model-repository option multiple times to included models from multiple repositories (Model Repository Doc)
@GuanLuo we do not yet support multiple cloud repositories for the same Triton server instance. This ticket should be left open
@CoderHam will proper error be returned currently if user specifies multiple cloud location?
@CoderHam will proper error be returned currently if user specifies multiple cloud location?
Good question, I don't think we have a check for that and I don't think an appropriate error will be returned.
@CoderHam do you mean triton cannot read from e.g. both aws s3 and google cloud?
Just to share, I tested with 2 x S3 buckets and 1 x local folder. Triton loaded all the models (in the 3 model repo folders) successfully. But if there are repeated model in the different model folder, there will be error.
I think you could use model namespacing to avoid collisions when using multiple model directories, including with cloud providers. See this test to see an example of how to enable model namespacing: https://github.com/triton-inference-server/server/blob/e787476ea3fefec5febfe7e382a9c2c6630c124b/qa/L0_model_namespacing/test.sh
