seldon-core icon indicating copy to clipboard operation
seldon-core copied to clipboard

Validate inputs from users automatically according to the field inputs defined in metadata

Open bydeath opened this issue 4 years ago • 3 comments

Model metadata of a seldondeployment https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/metadata.html#deployment-metadata allows us to set type and shape of inputs when we deploy a model.

I am thinking why not use inputs specified in metadata to validate inputs from users.

I am practicing the example examples/iris.html. It returns a 500 response when I send a invalid-shape input with curl. The response is unfriendly and not clear for who requests predicting. If there is a automatic validation according to inputs defined in deployment metadata, and give a clear response to tell users why they get a 500 error, it will be helpful for users to correct their inputs.

curl -s http://localhost:30550/seldon/seldon/seldon-deployment-example/api/v0.1/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[1,1,1]]}}'
\u003c!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"\u003e \u003ctitle\u003e500 Internal Server Error\u003c/title\u003e \u003ch1\u003eInternal Server Error\u003c/h1\u003e \u003cp\u003eThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\u003c/p\u003e

The log of container sklearn-iris-classifier of model pod is below when error occurs:

2020-07-17 02:52:33,235 - seldon_core.wrapper:log_exception:1892 - ERROR: Exception on /predict [POST] Traceback (most recent call last): File "/microservice/python/seldon_core/user_model.py", line 209, in client_predict return user_model.predict(features, feature_names, **kwargs) TypeError: predict() got an unexpected keyword argument 'meta' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/opt/conda/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/opt/conda/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise raise value File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/microservice/python/seldon_core/wrapper.py", line 54, in Predict user_model, requestJson, seldon_metrics File "/microservice/python/seldon_core/seldon_methods.py", line 108, in predict user_model, features, class_names, meta=meta File "/microservice/python/seldon_core/user_model.py", line 211, in client_predict return user_model.predict(features, feature_names) File "/microservice/IrisClassifier.py", line 9, in predict return self.model.predict_proba(X) File "/opt/conda/lib/python3.7/site-packages/sklearn/utils/metaestimators.py", line 119, in out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs) File "/opt/conda/lib/python3.7/site-packages/sklearn/pipeline.py", line 464, in predict_proba return self.steps[-1][-1].predict_proba(Xt) File "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py", line 1468, in predict_proba return super()._predict_proba_lr(X) File "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_base.py", line 321, in _predict_proba_lr prob = self.decision_function(X) File "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_base.py", line 287, in decision_function % (X.shape[1], n_features)) ValueError: X has 3 features per sample; expecting 4 2020-07-17 02:52:33,237 - werkzeug:_log:113 - INFO: 127.0.0.1 - - [17/Jul/2020 02:52:33] "POST /predict HTTP/1.1" 500 -

bydeath avatar Jul 17 '20 03:07 bydeath

@RafalSkolasinski , How do you think about it?

bydeath avatar Jul 17 '20 03:07 bydeath

I wish this was implemented, moreover, I thought this is what the model schema was needed for!

RadionBik avatar Jun 09 '21 11:06 RadionBik

@RadionBik @cliveseldon : Should this be added as an optional feature ? In my opinion, it might add latency to the request-response lifecycle but should be available as an option to enable while deploying.

rachitchauhan43 avatar Mar 29 '22 16:03 rachitchauhan43

Duplicate of https://github.com/SeldonIO/seldon-core/issues/3402

ukclivecox avatar Dec 05 '22 14:12 ukclivecox