Expected 2D array, got scalar array instead:\narray=nan.\nReshape your data either using array.reshape(-1, 1)

[nguyenthai0107]

/kind bug Hello guys, Im facing some issue when call POST with this example Here are some components

What steps did you take and what happened: [A clear and concise description of what the bug is.]

--- pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  namespace: kserve-test
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/ubuntu/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
  namespace: kserve-test
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

== InferenceService.yaml

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
  namespace: kserve-test
  annotations:
    "sidecar.istio.io/inject": "true"
spec:
  predictor:
    sklearn:
      name: kserve-container
      image: kserve/sklearnserver:v0.11.2
      storageUri: "pvc://task-pv-claim/model.joblib"

== output of kubectl get isvc -n kserve-test NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE sklearn-iris http://sklearn-iris.kserve-test.svc.cluster.local True 100 sklearn-iris-predictor-00001 43h

== im also deploy a pod inside cluster for debug when im ran curl -vvv -k http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris ( inside the pod ) It OK and get 200 status

/ #  curl -vvv -k http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris
*   Trying 10.x.x.x:80...
* Connected to sklearn-iris.kserve-test.svc.cluster.local (10.x.x.x) port 80 (#0)
> GET /v1/models/sklearn-iris HTTP/1.1
> Host: sklearn-iris.kserve-test.svc.cluster.local
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 38
< content-type: application/json
< date: Sun, 21 Jan 2024 06:52:34 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 6
< 
* Connection #0 to host sklearn-iris.kserve-test.svc.cluster.local left intact
{"name":"sklearn-iris","ready":"True"}/ # 

But when i ran for POST to predict which content iris-input.json

{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}

And i got issue with Status 500 error.

/ #  curl -vvv -k  POST http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris:predict -d @./iris-input.json
Warning: Couldn't read data from file "./iris-input.json", this makes an empty 
Warning: POST.
* Could not resolve host: POST
* Closing connection 0
curl: (6) Could not resolve host: POST
*   Trying 10.x.x.x:80...
* Connected to sklearn-iris.kserve-test.svc.cluster.local (10.x.x.x) port 80 (#1)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.svc.cluster.local
> User-Agent: curl/7.80.0
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< content-length: 211
< content-type: application/json
< date: Sun, 21 Jan 2024 06:55:31 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 6
< 
* Connection #1 to host sklearn-iris.kserve-test.svc.cluster.local left intact
{"error":"Expected 2D array, got scalar array instead:\narray=nan.\nReshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample."}/

What did you expect to happen: i expected output with status 200

/ #  curl -vvv -k http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris
*   Trying 10.x.x.x:80...
* Connected to sklearn-iris.kserve-test.svc.cluster.local (10.x.x.x) port 80 (#0)
> GET /v1/models/sklearn-iris HTTP/1.1
> Host: sklearn-iris.kserve-test.svc.cluster.local
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 38
< content-type: application/json
< date: Sun, 21 Jan 2024 06:52:34 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 6
< 
* Connection #0 to host sklearn-iris.kserve-test.svc.cluster.local left intact
{"name":"sklearn-iris","ready":"True"}/ # 

What's the InferenceService yaml: [To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]

Here is my isvc -oyaml

apiVersion: v1
items:
- apiVersion: serving.kserve.io/v1beta1
  kind: InferenceService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"serving.kserve.io/v1beta1","kind":"InferenceService","metadata":{"annotations":{"sidecar.istio.io/inject":"true"},"name":"sklearn-iris","namespace":"kserve-test"},"spec":{"predictor":{"sklearn":{"image":"kserve/sklearnserver:v0.11.2","name":"kserve-container","storageUri":"pvc://task-pv-claim/model.joblib"}}}}
      sidecar.istio.io/inject: "true"
    finalizers:
    - inferenceservice.finalizers
    generation: 1
    name: sklearn-iris
    namespace: kserve-test
  spec:
    predictor:
      model:
        image: kserve/sklearnserver:v0.11.2
        modelFormat:
          name: sklearn
        name: kserve-container
        resources: {}
        storageUri: pvc://task-pv-claim/model.joblib
  status:
    address:
      url: http://sklearn-iris.kserve-test.svc.cluster.local
    components:
      predictor:
        address:
          url: http://sklearn-iris-predictor.kserve-test.svc.cluster.local
        latestCreatedRevision: sklearn-iris-predictor-00001
        latestReadyRevision: sklearn-iris-predictor-00001
        latestRolledoutRevision: sklearn-iris-predictor-00001
        traffic:
        - latestRevision: true
          percent: 100
          revisionName: sklearn-iris-predictor-00001
        url: http://sklearn-iris-predictor.kserve-test.svc.cluster.local
    conditions:
    - lastTransitionTime: ""
      status: "True"
      type: IngressReady
    - lastTransitionTime: ""
      severity: Info
      status: "True"
      type: LatestDeploymentReady
    - lastTransitionTime: ""
      severity: Info
      status: "True"
      type: PredictorConfigurationReady
    - lastTransitionTime: ""
      status: "True"
      type: PredictorReady
    - lastTransitionTime: ""
      severity: Info
      status: "True"
      type: PredictorRouteReady
    - lastTransitionTime: ""
      status: "True"
      type: Ready
    - lastTransitionTime: ""
      severity: Info
      status: "True"
      type: RoutesReady
    modelStatus:
      copies:
        failedCopies: 0
        totalCopies: 1
      states:
        activeModelState: Loaded
        targetModelState: Loaded
      transitionStatus: UpToDate
    observedGeneration: 1
    url: http://sklearn-iris.kserve-test.svc.cluster.local
kind: List
metadata:
  resourceVersion: ""

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] I got models.joblib from here

Environment:

• Istio Version: 1.18.5
• Knative Version: 0.11.2
• KServe Version:0.11.2
• Kubeflow version: non-use
• Cloud Environment:[k8s_istio], baremetal and not use Cloud Privider, im also cannot config Istio for use LoadBalancer , so that used NodePort instead.
• Minikube/Kind version: none
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", 
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", 

• OS (e.g. from /etc/os-release): ubuntu-22.04 == here is log from pod/sklearn-iris-predictor-00001-deployment

2024-01-21 14:07:29.373 1 kserve ERROR [inference_error_handler():89] Exception:
Traceback (most recent call last):
  File "/sklearnserver/sklearnserver/model.py", line 61, in predict
    result = self._model.predict(instances)
  File "/prod_venv/lib/python3.9/site-packages/sklearn/svm/_base.py", line 818, in predict
    y = super().predict(X)
  File "/prod_venv/lib/python3.9/site-packages/sklearn/svm/_base.py", line 431, in predict
    X = self._validate_for_predict(X)
  File "/prod_venv/lib/python3.9/site-packages/sklearn/svm/_base.py", line 611, in _validate_for_predict
    X = self._validate_data(
  File "/prod_venv/lib/python3.9/site-packages/sklearn/base.py", line 604, in _validate_data
    out = check_array(X, input_name="X", **check_params)
2024-01-21 14:07:29.374 uvicorn.access INFO:     10.22.180.56:0 1 - "POST /v1/models/sklearn-iris%3Apredict HTTP/1.1" 500 Internal Server Error
  File "/prod_venv/lib/python3.9/site-packages/sklearn/utils/validation.py", line 932, in check_array
    raise ValueError(
ValueError: Expected 2D array, got scalar array instead:
array=nan.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/prod_venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/prod_venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/prod_venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/prod_venv/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/prod_venv/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/prod_venv/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/prod_venv/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/prod_venv/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/kserve/kserve/protocol/rest/v1_endpoints.py", line 76, in predict
    response, response_headers = await self.dataplane.infer(model_name=model_name,
  File "/kserve/kserve/protocol/dataplane.py", line 311, in infer
    response = await model(request, headers=headers)
  File "/kserve/kserve/model.py", line 122, in __call__
    else self.predict(payload, headers)
  File "/sklearnserver/sklearnserver/model.py", line 64, in predict
    raise InferenceError(str(e))
kserve.errors.InferenceError: Expected 2D array, got scalar array instead:
array=nan.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
2024-01-21 14:07:29.375 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.0021562576293945312
2024-01-21 14:07:29.375 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 0.002188000000000301

I just thining that issue releated with mode.joblib . So please take a look. Thank You

[nguyenthai0107]

Hello @yuzisun Could you pleasse take a look ? Thanks

[yuzisun]

@nguyenthai0107 did you add the application/json header ?

[amir-hoseyn77]

Hello, I have the same problem. It resolves by adding -H "Content-Type: application/json" to curl. I think without this header, fastapi can't parse the data. Final command changes to: (-k is not effective in HTTP)

curl -vvv -k -H "Content-Type: application/json" POST http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris:predict -d @./iris-input.json

[sivanantha321]

Closing this issue as per the above response.