Tianchu Zhao

Hi @tooptoop4 Were you able to make any progress on this issue, or is it still reoccurring?

> it still reoccurs but rare: roughly 1 in 20000 workflow runs. did u see the log in https://github.com/argoproj/argo-workflows/issues/12103#issuecomment-1784297997 ? Yes, but in my case, we are facing the same...

Hi @tooptoop4, we tried https://github.com/argoproj/argo-workflows/pull/12233 but it doesn't help in my case. I highly suspect the root cause in my case is due to https://github.com/argoproj/argo-workflows/issues/12997, we typically observe childnode missing...

I think I made a mistake in https://github.com/argoproj/argo-workflows/pull/12476 According to - https://argo-workflows.readthedocs.io/en/latest/default-workflow-specs/ - https://argo-workflows.readthedocs.io/en/latest/template-defaults/#configuring-templatedefaults-in-controller-level The priority should be: workflowDefault > template podSpecPatch > templateDefault. instead of current template podSpecPatch >...

>I think current priority is correct. workflowDefault is usually a common config, when I submit a specific workflow with different requirements, I will set a new config to override it....

The `test-corefunctional, minimal` was succeed before the last commit [2317852](https://github.com/argoproj/argo-workflows/pull/12960/commits/2317852e342db6995a3dac41b848288a983e0f19), The last commit only alters the `functional test`, the `test-corefunctional, minimal` failure is not an issue

I'm wondering how about having the numaflow python grpc client code as well, rather than just the server so that people can test their UDFs/UDSinks using the grpc client first...

Sorry, I was meant to add the commit hash for latest. Nothing to do with x.x.8 Updated

Screenshot for original issue 

Verify in the example logs ```yaml "test-workflow-kqxvv-113404891": { "boundaryID": "test-workflow-kqxvv-3314254127", "children": [ "test-workflow-kqxvv-1105926477" ], "displayName": "[0]", "finishedAt": null, "id": "test-workflow-kqxvv-113404891", "name": "test-workflow-kqxvv.onExit[0]", "nodeFlag": {}, "phase": "Running", "progress": "0/1", "startedAt": "2024-05-14T13:04:43Z",...