example-seldon icon indicating copy to clipboard operation
example-seldon copied to clipboard

Prediction Analytics Dashboard not showing metrics

Open jmdinis opened this issue 6 years ago • 0 comments

I deployed a keras mnist model using seldon core and I'm trying to monitor the model using seldon-core-analytics with grafana dashaboards. The cluster_monitoring dashboard looks fine but the Prediction Analytics doesn't find the my deployed model and all the panels are empty. I have installed kubeflow, the seldon-core and seldon-core-analytics in the same namespace and I have my model deployed in that same namespace.

After checking prometheus service logs in the same namespace I found the following message recurrently:

level=warn ts=2019-07-17T16:24:27.691294684Z caller=scrape.go:836 component="scrape manager" scrape_pool=kubernetes-pods target=http://192.168.49.198:16686/metrics msg="append failed" err=""INVALID" is not a valid start token"

Can you please advise on this?

Thanks.

jmdinis avatar Jul 17 '19 17:07 jmdinis