MLServer
MLServer copied to clipboard
SeldonIO
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
Here are a list of tickets that might affect alibi-explain runtime: Allow to specify channel dimension in data:https://github.com/SeldonIO/alibi/issues/487 AnchorImage will fail with != fp64 models:https://github.com/SeldonIO/alibi/issues/499 AnchorImage image_shape as List (needed...
Currently we return the output of explainers as v2 `BYTES` and downstream would need to know details about `alibi.Explanation` in order to decode and consume the response. Perhaps we need...
This might be a ticket on the alibi / core side but it is referenced here as it might affect mlserver batching. Currently we can only do one explanation for...
Currently, MLServer saves all the loaded models into an internal model registry (i.e. `dict`). In order to save resources when models are not being used, it could be useful to...
Similarly to what Triton server supports we could add support for binary data in payloads for http requests.
When adaptive batching is enabled, the `outputs` field should be forwarded down to the merged request to ensure that the model receives all the info.
Tempo currently has an implementation for an Insights Logger, which lets its internal runtime log metrics and payloads. Since this is relative low-level, it could be interesting to move this...
Hey there, Awesome framework. Would be nice to have a PyTorch Lightning or a Flash example (https://github.com/PyTorchLightning/lightning-flash). Best, T.C
Currently, `mlserver` relies on having `settings.json` and `model-settings.json` files present or falling back to environment variables. It would be good to also allow users to specify these flags directly through...
It would be great to add a PaddlePaddle inference runtime to MLServer. This could be modelled after the existing KFServing integration, which can be seen here: https://github.com/kubeflow/kfserving/blob/master/python/paddleserver/paddleserver/model.py