example-seldon
example-seldon copied to clipboard
Prediction Analytics Dashboard not showing metrics
I deployed a keras mnist model using seldon core and I'm trying to monitor the model using seldon-core-analytics with grafana dashaboards. The cluster_monitoring dashboard looks fine but the Prediction Analytics doesn't find the my deployed model and all the panels are empty. I have installed kubeflow, the seldon-core and seldon-core-analytics in the same namespace and I have my model deployed in that same namespace.
After checking prometheus service logs in the same namespace I found the following message recurrently:
level=warn ts=2019-07-17T16:24:27.691294684Z caller=scrape.go:836 component="scrape manager" scrape_pool=kubernetes-pods target=http://192.168.49.198:16686/metrics msg="append failed" err=""INVALID" is not a valid start token"
Can you please advise on this?
Thanks.