MLServer
MLServer copied to clipboard
Handle multiple models with custom endpoints
trafficstars
Following up from #167, there are a few things to take into account before adding support for custom endpoints across multiple models.
At the moment we just load the route as-is, without namespacing it with the model name. This means that loading multiple models could lead to clashes. For example, let's think of an inference runtime which registers a custom endpoint with path /foo. After loading 10 model instances using that runtime, it wouldn't be clear any more which one should be used to serve the custom endpoint with path /foo.
Some solutions that could be explored are:
- Disable custom endpoints when MMS is enabled. This would require adding the option to disable MMS.
- Namespace the custom path, e.g. registering
/v2/models/<model-name>/versions/<model-version>/fooinstead of just/foo. This could be easy to tackle, but it's not clear whether the custom endpoint would be as useful after adding the extra prefix (i.e. mainly as it would make it incompatible with legacy upstream services).