kubeflow-manifests
kubeflow-manifests copied to clipboard
Kubeflow integration with AWS managed Prometheus, Grafana and CW
Is your feature request related to a problem? Please describe. Logging and monitoring are two important aspect of MLOps platform. AWS distro of Kubeflow should have integration with aws managed Prometheus, grafana and cloudwatch(CW)
Describe the solution you'd like I believe we should order this in priority
- Logging: CW and Prometheus integration
- Monitoring: Grafana integration.
As a part of this issue, I would like to explore the viability of the option as well.
Perhaps, this can be option given to the user during installation stage as a config option. All services of AWS should be given as config options and defined by the user during start of installation and have an automatic script running to do do the installation. This will be super cool and will speed up the whole installation time and the user experience
I did Prometheus and Grafana integration separately using below
- Download the monitoring-core.yaml and monitoring-metrics-prometheus.yaml files from the Knative 0.18 release (https://github.com/knative/serving/releases/tag/v0.18.0)
- kubectl apply -f monitoring-core.yaml
- kubectl apply -f monitoring-metrics-prometheus.yaml
- kubectl apply -f grafana-virtual-service.yaml
- kubectl apply -f grafana-config-map.yaml
- kubectl apply -f models-web-app-aut-policy.yaml
- kubectl rollout restart deployment/grafana -n knative-monitoring
Login to Kubeflow and check if metrics are populating under Models
@Harikantipudi thanks for adding it. Is this only needed for serving or do we need something similar for notebook, tensorboard etc.?
@goswamig , the above setup is more for log monitoring like usage of CPU, Memory etc post your inference serving in Model UI for Kubeflow. However this can be extended for performance metrics and other monitoring aspects too
@goswamig Hello, is this issue stil WIP?
@AlexandreBrown This feature is complete and available in v1.6.1-aws-b1.0.0. https://awslabs.github.io/kubeflow-manifests/docs/add-ons/prometheus/guide/ https://awslabs.github.io/kubeflow-manifests/docs/add-ons/cloudwatch/guide/