kubeflow
[backend] Cannot get MLMD objects from Metadata store.
Environment
-
How did you deploy Kubeflow Pipelines (KFP)? Using Charmed kubeflow which is v1.10 on K8 v1.32
-
KFP version:
I am not sure of the exact version but hoping it should be >2.4.0 as I am using v1.10 kubeflow.
- KFP SDK version: 2.9.0
Steps to reproduce
Currently I do not have specific steps to reproduce the issue as it happened randomly
I am seeing an issue with one of my runs as shown in below snapshots.
As shown below, run_192613 is stuck and showing error message as “Cannot get MLMD objects from Metadata store.” When I clone the same run it is working fine. But the previous run is still stuck in the same error state.
I am not sure which exact logs need to be extracted from the system. Please do suggest which logs will be helpful in troubleshooting this issue so that I can capture the same.
Expected result
Run should not be stuck in error state
Materials and Reference
Impacted by this bug? Give it a 👍.
