Inferenceservice not responding to curl requests

[ajinkya933]

/kind bug

What steps did you take and what happened: I am following tutorial https://kserve.github.io/website/0.7/get_started/first_isvc/ and deploying first inferece service on KServe. I am stuck in last step where the service is deployed but I get no response in prediction. When I run performance test as shown in tutoial linked above I get error: Post "http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris:predict": dial tcp: lookup sklearn-iris.kserve-test.svc.cluster.local on 10.0.0.10:53: no such host

What did you expect to happen: Get response from inference server.

Anything else you would like to add: Error log after benchmarking step:

$ kubectl logs load-test948jk-k9d6c -n kserve-test
Requests      [total, rate, throughput]         30000, 500.02, 0.00
Duration      [total, attack, wait]             59.999s, 59.998s, 1.425ms
Latencies     [min, mean, 50, 90, 95, 99, max]  47.403µs, 1.563ms, 1.476ms, 1.853ms, 2.308ms, 4.46ms, 17.11ms
Bytes In      [total, mean]                     0, 0.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           0.00%
Status Codes  [code:count]                      0:30000  
Error Set:
Post "http://sklearn-iris.kserve-test.svc.cluster.local/v1/models/sklearn-iris:predict": dial tcp: lookup sklearn-iris.kserve-test.svc.cluster.local on 10.0.0.10:53: no such host

Environment:

• Istio Version: 1.11.1
• Knative Version: 0.25
• Kubernetes version: (use kubectl version): 1.21

[yuzisun]

@ajinkya933 did the inference service get into ready status? run kubectl get isvc sklearn-iris -oyaml should give me more debugging info.

[ajinkya933]

I was using a load balancer and did not mention auth service cookie in header so that was causing this issue.