[frontend] Pipeline status unknown, cannot fetch

[tomaszstachera]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
• https://awslabs.github.io/kubeflow-manifests/release-v1.7.0-aws-b1.0.3/docs/deployment/vanilla/guide/
• KFP version:
• build version dev_local (https://awslabs.github.io/kubeflow-manifests/release-v1.7.0-aws-b1.0.3/docs/deployment/vanilla/guide/)

Steps to reproduce

Run a lot of pipelines (few thousands for few months), also I did a mysql db migration from one PVC to another due to full disk.

Expected result

Proper pipeline status and graphs shown.

Materials and Reference

Screenshots:

Other pipelines started from recurring run show proper status. Few started manually show unknown status.

Impacted by this bug? Give it a 👍.

[HumairAK]

@tomaszstachera thanks, can you peak at the logs from the server/ui/persistent-agent pods when the failures occur, do you notice any errors?

[tomaszstachera]

Issue disappeared without any changes. This can be closed.