Merge kubernetes pod objects inplace

[dirrao]

What happened

In the Kubernetes executor, Merge Kubernetes Pod objects (base_worker_pod with 3 containers and 1 init container) is taking a significant amount of time (> 500 ms) due to the recursive deep copy.

What you think should happen instead

We are constructing the base_worker_pod, dynamic_pod, and executor_config_pod for every task. So, we don't need to retain these pod objects. We can merge these pod objects in place and return the merged pod.

I agree with fact that the immutability brings advantages. Even if we want to retain the pod objects, clone them before calling construct_pod/reconcile_pods function.

Close: #37131

[dirrao]

@dirrao in terms of numbers, how much improvement does this bring to the table? Just curious.

Btw, wondering if there's any risk associated here with this.

I have seen the 3x improvement in the scheduler performance. Check the same in the #37131.

[dirrao]

@hussein-awala / @jedcunningham This is breaking with recent changes from Kubernetes job operators. I would like to hear from others if it make sense for Kubernetes executor. I will add flag to enable it for Kubernetes executor and disable it for operators.

[potiuk]

what do you mean by breaking IMHO it's just broken, not breaking - because it's impossible to foresee how one pod executing will impact another one, and since you have no influence on that, this is indeterministic. Which trades of performance with predictability . Or am I wrong?

[dirrao]

what do you mean by breaking IMHO it's just broken, not breaking - because it's impossible to foresee how one pod executing will impact another one, and since you have no influence on that, this is indeterministic. Which trades of performance with predictability . Or am I wrong?

When I made these changes sometime back when there was no Kubernetes job operator. These Kubernetes operators started using the Kubernetes utilities. Immutability is mandatory for Kubernetes operators not for Kubernetes executor. Right now the proposed changes don't distinguish between operators and executor path. I can add a flag to Kubernetes utilities to enable/disable in-place pod spec merging.

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[dirrao]

@jedcunningham / @hussein-awala We've observed degraded performance in pod construction in version 2.9.2 compared to 2.3.3. Would you consider an in-place merge to address this?