predict() got an unexpected keyword argument 'meta'

[yantriks-edi-bice]

/kind bug

What steps did you take and what happened:

Deployed Kubeflow 1.0 on GCP and worked through xgboost-high-level-apis.ipynb. All went well until the endpoint.predict_nparray(test_X) step which generated following error

---

JSONDecodeError Traceback (most recent call last) in 1 #(train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X) 3 test_X[1,]

~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self):

/usr/lib/python3.6/json/init.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder

/usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s):

/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

And on the GCP project error side another error which coincides in time but have not been able to

{ insertId: "u0az0ar5xu86c13t7"
labels: { k8s-pod/fairing-deployer: "serving"
k8s-pod/fairing-id: "83b1d074-5d82-11ea-a6ca-3a811b82dfb5"
k8s-pod/pod-template-hash: "7946c4cd6c"
} logName: "projects/saas-ml-dev/logs/stderr"
receiveTimestamp: "2020-03-03T21:34:05.657934723Z"
resource: { labels: { cluster_name: "kf-tst"
container_name: "model"
location: "us-central1-a"
namespace_name: "kubeflow-edi-bice"
pod_name: "fairing-deployer-5gzht-7946c4cd6c-vzvmj"
project_id: "saas-ml-dev"
} type: "k8s_container"
} severity: "ERROR"
textPayload: "Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/seldon_core/user_model.py", line 199, in client_predict return user_model.predict(features, feature_names, **kwargs) TypeError: predict() got an unexpected keyword argument 'meta' "
timestamp: "2020-03-03T21:34:04.298606981Z"
}

What did you expect to happen:

Prediction result printed to notebook output

Anything else you would like to add:

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"):

Version: 0.7.1

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):

1.0

• Minikube version:
• Kubernetes version: (use kubectl version):

1.14.10-gke.22

• OS (e.g. from /etc/os-release):

GCP Container OS

NOTE: If you are using fair from master, please provide us the git commit hash.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.99

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[chantikummari]

facing same issue with azure also:

JSONDecodeError Traceback (most recent call last) in 1 (train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X) 3 #test_X[1,] 4 #endpoint.predict(test_X)

~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self):

/usr/lib/python3.6/json/init.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder

/usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s):

/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[chaitanyak52]

I see the same issue on azure

---

JSONDecodeError Traceback (most recent call last) in 1 (train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X)

~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self):

/usr/lib/python3.6/json/init.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder

/usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s):

/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jinchihe]

Anyone interested in solving this problem? Thanks

[chantikummari]

Anyone interested in solving this problem? Thanks

Yes

[jinchihe]

@chantikummari Did you tried this with new kubeflow-fairing? the example is executed and passed in CI testing. Thanks.

[chantikummari]

@chantikummari Did you tried this with new kubeflow-fairing? the example is executed and passed in CI testing. Thanks.

I didn't try yet. We skipped that fairing feature from our project. I will definitely check it if any requirement comes related to it.

[jinchihe]

So I suggest use latest kubeflow-fairing, and use new the example, the example is updated, such as in the PR: https://github.com/kubeflow/fairing/pull/335

[achal-wwt]

@chantikummari, @jinchihe I was facing the same issue while trying Fairing on kubeflow cluster on AWS. I was trying to deploy my Fashion MNIST example using Fairing. My error got resolved when I put in feature_names=None in my predict function. My function call was - def predict(self, X, feature_names=None): # code to predict

[Juggernaut1997]

@achal-wwt Hey can you share how did you deploy it. Because I am facing the same issue and not getting how to resolve it

[achal-wwt]

@Juggernaut1997 I currently don't have the code available to share but if you are facing issue while using the trained model i.e. while getting prediction . Just make sure that you pass in feature_names as an argument in the predict function even though you are not using the argument.

[Juggernaut1997]

@achal-wwt Hi thanks for the reply, sorry i didn't respond early. But i am still facing the same issue.

[Anh-BK]

I realize that when I run seldon-core in the container, I occur the error like above. However, when I run with cpu, the error is gone.