pipelines icon indicating copy to clipboard operation
pipelines copied to clipboard

[bug] Considerable delay in showing steps in Kubeflow Pipelines UI after navigating to Runs

Open rafalk0 opened this issue 3 years ago • 2 comments

What steps did you take

Create a pipeline with working ML algorithm.

What happened:

We encountered considerable delay (lasting more than 2 minutes) in showing steps in Kubeflow UI after navigating to Runs. Initially, after opening Runs, Kubeflow Pipelines UI shows only a blank window. The algorithm's steps in Kubeflow Pipelines appear only after these 2 requests: ml_metadata.MetadataStoreService/GetContextByTypeAndName fail. It seems that it's rather a core Kubeflow's internal bug, similar to the one reported here: https://github.com/kubeflow/pipelines/issues/3763. This issue affects only the Kubeflow UI and does not affect the algorithms' deployments nor operations. We can still access pipelines details by using an external software as UI.

Kubeflow_140_170_working_slow

What did you expect to happen:

We didn't expect any significant delay.

Environment:

  • How do you deploy Kubeflow Pipelines (KFP)? Kubeflow Pipelines as part of a full Kubeflow deployment. Kubeflow itself is installed using kustomize, according to the documentation (agnostic environment).
  • KF version: 1.4.0
  • KFP version: 1.7.0
  • KFP SDK version: 1.8.11

Anything else you would like to add:

What would be the estimated time of resolution for this issue?

Labels


Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

rafalk0 avatar Jun 14 '22 08:06 rafalk0

Could you check the resource usage in your cluster, and see if any limit was reached? You may also consider upgrading to 1.8.

Linchin avatar Jun 30 '22 22:06 Linchin

@Linchin No limit was reached when it comes to resource usage.

rafalk0 avatar Jul 18 '22 10:07 rafalk0

Closing this issue. No activity for more than a year.

/close

rimolive avatar Apr 03 '24 16:04 rimolive

@rimolive: Closing this issue.

In response to this:

Closing this issue. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

google-oss-prow[bot] avatar Apr 03 '24 16:04 google-oss-prow[bot]