kubeflow-manifests
kubeflow-manifests copied to clipboard
awslabs
Missing metrics data in RDS while using KFPv2
Describe the bug Vishnu Paspunoor 12:33 AM Hi all, I'm using kubeflow with rds, s3 and cognito. Any idea how to upload the pipeline metrics into RDS in KFP SDK v2? I could see the metrics in the run_metrics table when I tried v1 but there are no entries while using scalar metrics in v2. v1 example: https://www.kubeflow.org/docs/components/pipelines/sdk/pipelines-metrics/ v2 example: https://www.kubeflow.org/docs/components/pipelines/sdk-v2/python-function-components/#example-python-function-based-component Tried this too https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#scalar-metrics Additionally, the run outputs tab doesn't show any data in v2
Steps To Reproduce
Expected behavior A clear and concise description of what you expected to happen.
Environment
- Kubernetes version
- Using EKS (yes/no), if so version?
- Kubeflow version
- AWS build number
- AWS service targeted (S3, RDS, etc.)
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.
Here are some additional details
Steps To Reproduce
Execute these examples in kubeflow and query the run_metrics table in the mlpipeline database on RDS mysql
Expected behavior
Run metrics are expected to be present in the run_metrics table
Environment
Kubernetes version: 1.21 Using EKS (yes/no), if so version? yes 1.21 Kubeflow version: 1.51 AWS build number AWS service targeted (S3, RDS, etc.): RDS Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.
@vishnuisu I will attempt to re-create this behavior and get back to you asap
@vishnuisu Based off of this issue metrics are not yet supported in v2 compatibility mode, as well as the run output being empty. https://github.com/kubeflow/pipelines/issues/6132 will reach out to pipeline folks and see if anything has changed. However you should be able to still see your metrics being sent to rds if you check your Artifacts and ArtifactProperty tables in your database.