kubeflow
[backend] Performance issue: ScheduledWorkflow is taking significant amount of etcd storage
Environment
- How did you deploy Kubeflow Pipelines (KFP)? Full Kuebflow deployment using manifests
- KFP version: 2.0.0b6
- KFP SDK version: 2.0.0b6
Steps to reproduce
We have around 125 recurring runs within a single namespace. After a few months of historical runs, we have started seeing performance issues in the k8s cluster.
After digging deeper, we found that we are seeing timeouts in the calls to etcd. When we checked the etcd database for objects we found that one particular namespace which has 125 recurring runs is taking 996MB of etcd space
some data to look at:
Entries by 'KEY GROUP' (total 1.6 GB):
+----------------------------------------------------------------------------------------------------------+--------------------------------+--------+
| KEY GROUP | KIND | SIZE |
+----------------------------------------------------------------------------------------------------------+--------------------------------+--------+
| /registry/kubeflow.org/scheduledworkflows/<namespace1> | ScheduledWorkflow | 996 MB |
| /registry/kubeflow.org/scheduledworkflows/<namespace2> | ScheduledWorkflow | 211 MB |
| /registry/kubeflow.org/scheduledworkflows/<namespace3> | ScheduledWorkflow | 118 MB |
.....
namespace1 has 123 recurring runs namespace2 has 40 recurring runs namespace3 has 63 recurring runs
Expected result
Looks like we are storing a lot of unnecessary information in the ScheduledWorkflow object, which eventually is taking space in etcd database resulting in all the performance issues
Materials and Reference
Impacted by this bug? Give it a 👍.
Hi @deepk2u! It may be due to insufficient resource provisioning or the lack of etc maintenance (see here). How longs did it take for you to reach these numbers?
It's an eks cluster. We connected with AWS support and they are maintaining the cluster and running defragmentation and doing all kinds of maintenance for etcd.
the oldest object I have on the list is from 14th July 2022.
Can check how large are your pipeline manifests used in these recurring runs?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.
@kuldeepjain: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
![google-oss-prow[bot] avatar](https://avatars.githubusercontent.com/in/143327?v=4 "Published by a pub.dev verified publisher"){kind=small}
@deepk2u: Reopened this issue.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@rimolive: Closing this issue.
In response to this:
Closing this issue. No activity for more than a year.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
/reopen
We found this issue in KFP 2.0.5. We'll work on a pruning mechanism for pipeline run k8s objects.
@rimolive: Reopened this issue.
In response to this:
/reopen
We found this issue in KFP 2.0.5. We'll work on a pruning mechanism for pipeline run k8s objects.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.
/reopen
I am seeing this issue in KFP 2.2.0. Our cluster has ~50 ScheduledWorkflows and we are seeing ~400MB of bytes written to ETCD every 10 minutes for the /registry/kubeflow.org/scheduledworkflows prefix.
The culprit seems to the heartbeat status updates, e.g.:
Status:
Conditions:
Last Heartbeat Time: 2024-09-19T11:16:33Z
Last Transition Time: 2024-09-19T11:16:33Z
Message: The schedule is disabled.
Reason: Disabled
Status: True
Type: Disabled
These status updates mean the SWF objects fail to reconcile, resulting in the following reconciliation loop:
- SWF is added to controller work queue
- Controller processes the SWF and updates the status heartbeat and transition time to current time.
- Object is re-written to ETCD and resourceVersion is updated.
- Controller event handler re-adds the SWF to the work queue.
This reconciliation loop occurs every 10 seconds for every SWF on the cluster (note: the reason it's 10s and not 1s is because of the controller's default queue backoff, so events are always queued for a minimum of 10s).
How is the heartbeat time and transition time used in Kubeflow? If they are not used, then one possible fix here would be to remove them from the status block.
cc @droctothorpe
@demarna1: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
I am seeing this issue in KFP 2.2.0. Our cluster has ~50 ScheduledWorkflows and we are seeing ~400MB of bytes written to ETCD every 10 minutes for the
/registry/kubeflow.org/scheduledworkflowsprefix.The culprit seems to the heartbeat status updates, e.g.:
Status: Conditions: Last Heartbeat Time: 2024-09-19T11:16:33Z Last Transition Time: 2024-09-19T11:16:33Z Message: The schedule is disabled. Reason: Disabled Status: True Type: DisabledThese status updates mean the SWF objects fail to reconcile, resulting in the following reconciliation loop:
- SWF is added to controller work queue
- Controller processes the SWF and updates the status heartbeat and transition time to current time.
- Object is re-written to ETCD and resourceVersion is updated.
- Controller event handler re-adds the SWF to the work queue.
This reconciliation loop occurs every 10 seconds for every SWF on the cluster (note: the reason it's 10s and not 1s is because of the controller's default queue backoff, so events are always queued for a minimum of 10s).
How is the heartbeat time and transition time used in Kubeflow? If they are not used, then one possible fix here would be to remove them from the status block.
cc @droctothorpe
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@droctothorpe: Reopened this issue.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.