models-web-app
models-web-app copied to clipboard
kserve
Metrics not showing
I'm running into an issue with the metrics tab of the web app. Following the documentation (https://www.kubeflow.org/docs/external-add-ons/kserve/webapp/#metrics), I can confirm that Grafana is accessible and the metrics tab is displayed.
Metrics for an InferenceService are also successfully picked up and shown in the Grafana Dashboard. However, these metrics are not displayed in the the web app directly:
CPU and Memory metrics eventually show up in the web app but with a noticeable delay.
The same setup was working fine in Kubeflow 1.5 (KServe 0.7). The issue only occurs with Kubeflow 1.6 (KServe 0.8).
Looking at the network traffic, I can see that the web app sends requests to the grafana endpoint. For example (shortened):
https://example.com/grafana/.../knative-serving-revision-http-requests?var-namespace=undefined&var-configuration=mymodel-predictor-default
Notice how the var-namespace parameter is set to undefined which leads to an empty response from grafana. The link is working correctly when properly setting the namespace.
The issue seems to be the order of execution:
https://github.com/kserve/models-web-app/blob/a3a8e80492a81cfe307299efd926513663f60146/frontend/src/app/pages/server-info/metrics/metrics.component.ts#L18-L31
The graphs are generated in the @Input() of the InferenceServiceStatus. Unfortunately, the @Input() of the namespace seems to be executed afterwards. Therefore, the namespace is undefined for the generation of the graphs.
I'm not sure how this issue was introduced. There were no changes to the relevant code in quite a while. Is anyone able to confirm?
Closing as completed since https://github.com/kserve/models-web-app/pull/70 has been merged.