MLServer
MLServer copied to clipboard
SeldonIO
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
After updating images to new versions of MLServer (version `1.3.0.dev4`),`batch_request_queue` and `parallel_request_queue` are not showing up on my K8S prometheus when deploying them on SeldonDeployments V1. Other parameters like `model_infer_request_success`...
When using custom environments, the model can't be even imported within the main process. Therefore, the custom handlers hook is not able to inspect the model to look for custom...
Hi, I'm trying to improve throughput of my server which was running by MLServer, and I got to know that I can set 'parallel_workers > 1' to enable parallel. Hence...
There is currently no inbuilt way to validate an input against its defined schema. For example, this schema defines an input array named `array` with shape `[-1, 1, 28, 28]`,...
Hi, I'm using MLServer with KServe, and found that the proto descriptor in grpc has a collision between them: ``` File ~/.cache/pypoetry/virtualenvs/example-mlflow-lZ2hGP5g-py3.10/lib/python3.10/site-packages/mlserver/__init__.py:2 1 from .version import __version__ ----> 2 from...
Feature request - Since doing an optimization step before the deep learning model is becoming very common in machine learning deployment, out-of-the-box support in MLServer could be beneficial, some examples...
The `starlette_exporter` middleware we use in the REST server seems to cap buckets at 10s (which matches with the default buckets used in Prom: https://github.com/prometheus/client_python/blob/4f994ece6dcfd1905726d18e2a6899cc4474ac3d/prometheus_client/metrics.py#L544). In some cases, requests' latency...
As a follow up to #1020, it would be good to support other types of metrics under the "simplified" interface (e.g `Counter`, `Summary`, `Gauge`, etc.). Or, alternatively, under our own...
If an explainer (e.g. anchors) does not allow batched explanation, then we could in the runtime loop over each element in the batch and conduct explanation of each item. This...
Currently we allow users to pass in runtime explain parameters via setting `explain_parameters` in the `parameters` field of the explanation payload. This allows only basic types. In some cases the...