argo-workflows Sprig templating not respected

Pre-requisites

[x] I have double-checked my configuration
[x] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[x] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Seems that at some point between 3.6.5 and 3.6.12 sprig templating seems to no longer be respected.

Version(s)

3.6.12+

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

metadata:
  name: regression-test
spec:

  entrypoint: argosay
  templates:
    - name: argosay
      script:
        env:
          - name: MESSAGE
            value: "{{=sprig.dig('some', 'label', 'io/creator-email, nil, workflow.labels) ?? sprig.dig('workflows', 'argoproj', 'io/creator-email, 'undefined', workflow.labels)}}"
        image: alpine:latest
        command:
          - '/bin/sh'
        source: |
          #!/usr/bin/env bash
          set -eo pipefail
          
          echo "$MESSAGE"

in 3.6.5 the workflow prints:

undefined when some.label.io/creator-email and workflows.argoproj.io/creator-email are both not set in labels

dmarquez.at.splunk.com when I submit through the UI

value of label some.label.io/creator-email when created through internal automation

In 3.6.12 the workflow prints:

{{=sprig.dig('some', 'label', 'io/workflow-template', nil, workflow.labels) ?? sprig.dig('workflows', 'argoproj', 'io/workflow-template', 'undefined', workflow.labels)}}

Logs from the workflow controller

3.6.5 logs:


~ % kubectl logs -n wfe argo-workflow-controller-wfe-565d6b74c-lmzs8 | grep regression-test-2
time="2025-12-02T14:24:17.419Z" level=info msg="Processing workflow" Phase= ResourceVersion=1845256500 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.420Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.420Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.426Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=0 workflow=regression-test-2
time="2025-12-02T14:24:17.426Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.426Z" level=info msg="Updated phase  -> Running" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.426Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.426Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.426Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.426Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.426Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.427Z" level=debug msg="Initializing node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.427Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.427Z" level=info msg="Pod node regression-test-2 initialized Pending" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.427Z" level=debug namespace=<namespace> needLocation=true workflow=regression-test-2
time="2025-12-02T14:24:17.427Z" level=warning msg="couldn't get boundaryTemplate through nodeName regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.427Z" level=debug msg="Creating Pod: regression-test-2 (regression-test-2)" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.642Z" level=info msg="Created pod: regression-test-2 (regression-test-2)" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.642Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.642Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17Z" level=info msg="add pod event" pod=regression-test-2
time="2025-12-02T14:24:17.643Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"wfe.splunk8s.io/host\":\"cw-wfe.playground.svc.splunk8s.io\",\"workflows.argoproj.io/archive-strategy\":\"always\",\"workflows.argoproj.io/completed\":\"false\",\"workflows.argoproj.io/phase\":\"Running\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"splunk8s.io/burst\",\"operator\":\"Exists\"}]}]}}},\"securityContext\":{\"runAsNonRoot\":true,\"runAsUser\":510,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"templateDefaults\":{\"container\":{\"name\":\"\",\"resources\":{},\"volumeMounts\":[{\"mountPath\":\"/etc/meow\",\"name\":\"meow-config\"}]},\"inputs\":{},\"metadata\":{\"annotations\":{\"iam.amazonaws.com/role\":\"arn:aws:iam::841210586048:role/wfe-cyclops-pdx10-play-shirley-s3-role\",\"proxy.istio.io/config\":\"{ \\\"holdApplicationUntilProxyStarts\\\": true }\",\"sidecar.istio.io/proxyCPU\":\"100m\",\"sidecar.istio.io/proxyCPULimit\":\"100m\",\"sidecar.istio.io/proxyMemory\":\"150Mi\",\"sidecar.istio.io/proxyMemoryLimit\":\"150Mi\",\"traffic.sidecar.istio.io/excludeOutboundIPRanges\":\"100.64.0.1/0\",\"workflows.argoproj.io/kill-cmd-istio-proxy\":\"[\\\"pilot-agent\\\", \\\"request\\\", \\\"POST\\\", \\\"quitquitquit\\\"]\"}},\"outputs\":{},\"script\":{\"name\":\"\",\"resources\":{},\"source\":\"\",\"volumeMounts\":[{\"mountPath\":\"/etc/meow\",\"name\":\"meow-config\"}]},\"volumes\":[{\"configMap\":{\"name\":\"meow-config\",\"optional\":true},\"name\":\"meow-config\"}]},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"splunk8s.io/burst\",\"operator\":\"Exists\"}],\"ttlStrategy\":{\"secondsAfterCompletion\":300}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"bucket\":\"wfe-cyclops-pdx10-play-shirley-logs\",\"endpoint\":\"s3.amazonaws.com\",\"insecure\":false,\"keyFormat\":\"{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}\",\"region\":\"us-west-2\",\"useSDKCreds\":true}},\"default\":true},\"nodes\":{\"regression-test-2\":{\"displayName\":\"regression-test-2\",\"finishedAt\":null,\"id\":\"regression-test-2\",\"name\":\"regression-test-2\",\"phase\":\"Pending\",\"startedAt\":\"2025-12-02T14:24:17Z\",\"templateName\":\"argosay\",\"templateScope\":\"local/regression-test-2\",\"type\":\"Pod\"}},\"phase\":\"Running\",\"startedAt\":\"2025-12-02T14:24:17Z\"}}"
time="2025-12-02T14:24:17Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:17.654Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=1845256575 workflow=regression-test-2
time="2025-12-02T14:24:17.655Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845256575 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.656Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=0 workflow=regression-test-2
time="2025-12-02T14:24:17.656Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.656Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Pending new.progress=0/1 nodeID=regression-test-2 old.message= old.phase=Pending old.progress=0/1 workflow=regression-test-2
time="2025-12-02T14:24:17.656Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.657Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.657Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.657Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.657Z" level=debug msg="Executing node regression-test-2 of Pod is Pending" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.657Z" level=debug msg="Skipped pod regression-test-2 (regression-test-2) creation: already exists" namespace=<namespace> podPhase=Pending workflow=regression-test-2
time="2025-12-02T14:24:17.657Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.657Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.658Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"regression-test-2\":{\"hostNodeName\":\"ip-10-120-11-50.us-west-2.compute.internal\"}}}}"
time="2025-12-02T14:24:17Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:17.670Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=1845256580 workflow=regression-test-2
time="2025-12-02T14:24:17.670Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845256580 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.671Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=0 workflow=regression-test-2
time="2025-12-02T14:24:17.671Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=info msg="node changed" namespace=<namespace> new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=regression-test-2 old.message= old.phase=Pending old.progress=0/1 workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.672Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.672Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:17.672Z" level=debug msg="Executing node regression-test-2 of Pod is Pending" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=debug msg="Skipped pod regression-test-2 (regression-test-2) creation: already exists" namespace=<namespace> podPhase=Pending workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.672Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:17.673Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"regression-test-2\":{\"message\":\"PodInitializing\"}}}}"
time="2025-12-02T14:24:17.688Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=1845256583 workflow=regression-test-2
time="2025-12-02T14:24:19Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:20Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:22Z" level=info msg="insignificant pod change" key=<namespace>/regression-test-2
time="2025-12-02T14:24:23Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:24Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:25Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:26Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:27.644Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845256583 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{regression-test-2  <namespace>  9264e6c5-a98a-42de-a418-98e520fea66f 1845257042 1 2025-12-02 14:24:26 +0000 UTC <nil> <nil> map[workflows.argoproj.io/report-outputs-completed:false workflows.argoproj.io/workflow:regression-test-2] map[] [{argoproj.io/v1alpha1 Workflow regression-test-2 afefe2df-04d0-4854-a658-e8778a90afeb <nil> <nil>}] [] [{argoexec Update argoproj.io/v1alpha1 2025-12-02 14:24:26 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"afefe2df-04d0-4854-a658-e8778a90afeb\\\"}\":{}}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:nil,Progress:,},}" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="task result name:\nregression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="Marking task result incomplete regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Running new.progress=0/1 nodeID=regression-test-2 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.645Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.645Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.645Z" level=debug msg="Executing node regression-test-2 of Pod is Running" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.645Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.646Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"regression-test-2\":{\"message\":null,\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"regression-test-2\":false}}}"
time="2025-12-02T14:24:27.682Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=1845257097 workflow=regression-test-2
time="2025-12-02T14:24:27.682Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"<namespace>\", Name:\"regression-test-2\", UID:\"afefe2df-04d0-4854-a658-e8778a90afeb\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"1845257097\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node regression-test-2"
time="2025-12-02T14:24:27.683Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845257097 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.683Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-2
time="2025-12-02T14:24:27.683Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{regression-test-2  <namespace>  9264e6c5-a98a-42de-a418-98e520fea66f 1845257042 1 2025-12-02 14:24:26 +0000 UTC <nil> <nil> map[workflows.argoproj.io/report-outputs-completed:false workflows.argoproj.io/workflow:regression-test-2] map[] [{argoproj.io/v1alpha1 Workflow regression-test-2 afefe2df-04d0-4854-a658-e8778a90afeb <nil> <nil>}] [] [{argoexec Update argoproj.io/v1alpha1 2025-12-02 14:24:26 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"afefe2df-04d0-4854-a658-e8778a90afeb\\\"}\":{}}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:nil,Progress:,},}" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.683Z" level=debug msg="task result name:\nregression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.683Z" level=debug msg="Marking task result incomplete regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.683Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.684Z" level=info msg="node unchanged" namespace=<namespace> nodeID=regression-test-2 workflow=regression-test-2
time="2025-12-02T14:24:27.684Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.684Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.684Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.684Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:27.684Z" level=debug msg="Executing node regression-test-2 of Pod is Running" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.684Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:27.684Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:29Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:30Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:36Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:37.684Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845257097 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{regression-test-2  <namespace>  9264e6c5-a98a-42de-a418-98e520fea66f 1845257172 2 2025-12-02 14:24:26 +0000 UTC <nil> <nil> map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:regression-test-2] map[] [{argoproj.io/v1alpha1 Workflow regression-test-2 afefe2df-04d0-4854-a658-e8778a90afeb <nil> <nil>}] [] [{argoexec Update argoproj.io/v1alpha1 2025-12-02 14:24:29 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"afefe2df-04d0-4854-a658-e8778a90afeb\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,SessionTokenSecret:nil,},Key:<namespace>/regression-test-2/regression-test-2/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=debug msg="task result name:\nregression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=debug msg="Marking task result complete regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=info msg="task-result changed" namespace=<namespace> nodeID=regression-test-2 workflow=regression-test-2
time="2025-12-02T14:24:37.684Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:37.685Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Running new.progress=0/1 nodeID=regression-test-2 old.message= old.phase=Running old.progress=0/1 workflow=regression-test-2
time="2025-12-02T14:24:37.685Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.685Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.685Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.685Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.685Z" level=debug msg="Executing node regression-test-2 of Pod is Running" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.685Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.685Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.685Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"regression-test-2\":{\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"<namespace>/regression-test-2/regression-test-2/main.log\"}}],\"exitCode\":\"0\"}}},\"taskResultsCompletionStatus\":{\"regression-test-2\":true}}}"
time="2025-12-02T14:24:37Z" level=info msg="cleaning up pod" action=terminateContainers key=<namespace>/regression-test-2/terminateContainers namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:37Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:37.699Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=1845257472 workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845257472 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{regression-test-2  <namespace>  9264e6c5-a98a-42de-a418-98e520fea66f 1845257172 2 2025-12-02 14:24:26 +0000 UTC <nil> <nil> map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:regression-test-2] map[] [{argoproj.io/v1alpha1 Workflow regression-test-2 afefe2df-04d0-4854-a658-e8778a90afeb <nil> <nil>}] [] [{argoexec Update argoproj.io/v1alpha1 2025-12-02 14:24:29 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"afefe2df-04d0-4854-a658-e8778a90afeb\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,SessionTokenSecret:nil,},Key:<namespace>/regression-test-2/regression-test-2/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=debug msg="task result name:\nregression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=debug msg="Marking task result complete regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.700Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:37.701Z" level=info msg="node unchanged" namespace=<namespace> nodeID=regression-test-2 workflow=regression-test-2
time="2025-12-02T14:24:37.701Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.701Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.701Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.701Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:37.701Z" level=debug msg="Executing node regression-test-2 of Pod is Running" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.701Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37.701Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:37Z" level=info msg="cleaning up pod" action=terminateContainers key=<namespace>/regression-test-2/terminateContainers namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:37Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:38Z" level=info msg="update pod event" pod=regression-test-2
time="2025-12-02T14:24:47.699Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1845257472 namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{regression-test-2  <namespace>  9264e6c5-a98a-42de-a418-98e520fea66f 1845257172 2 2025-12-02 14:24:26 +0000 UTC <nil> <nil> map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:regression-test-2] map[] [{argoproj.io/v1alpha1 Workflow regression-test-2 afefe2df-04d0-4854-a658-e8778a90afeb <nil> <nil>}] [] [{argoexec Update argoproj.io/v1alpha1 2025-12-02 14:24:29 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"afefe2df-04d0-4854-a658-e8778a90afeb\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,SessionTokenSecret:nil,},Key:<namespace>/regression-test-2/regression-test-2/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="task result name:\nregression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="Marking task result complete regression-test-2" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="Skipping artifact GC" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Succeeded new.progress=0/1 nodeID=regression-test-2 old.message= old.phase=Running old.progress=0/1 workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="Evaluating node regression-test-2: template: *v1alpha1.WorkflowStep (argosay), boundaryID: " namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.700Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:47.700Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:47.700Z" level=debug msg="Getting the template by name: argosay" base="*v1alpha1.Workflow (namespace=<namespace>,name=regression-test-2)" tmpl="*v1alpha1.WorkflowStep (argosay)"
time="2025-12-02T14:24:47.701Z" level=debug msg="Node regression-test-2 already completed" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=debug msg="Task results completion status: map[regression-test-2:true]" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=info msg="Updated phase Running -> Succeeded" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=info msg="Marking workflow completed" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=info msg="Marking workflow as pending archiving" namespace=<namespace> workflow=regression-test-2
time="2025-12-02T14:24:47.701Z" level=debug msg="Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2025-12-02T14:24:47Z\",\"nodes\":{\"regression-test-2\":{\"finishedAt\":\"2025-12-02T14:24:35Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":1,\"ephemeral-storage\":0,\"memory\":56}}},\"phase\":\"Succeeded\"}}"
time="2025-12-02T14:24:47.713Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Succeeded resourceVersion=1845257786 workflow=regression-test-2
time="2025-12-02T14:24:47.713Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"<namespace>\", Name:\"regression-test-2\", UID:\"afefe2df-04d0-4854-a658-e8778a90afeb\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"1845257786\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node regression-test-2"
time="2025-12-02T14:24:47Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:47.729Z" level=info msg="archiving workflow" namespace=<namespace> uid=afefe2df-04d0-4854-a658-e8778a90afeb workflow=regression-test-2
time="2025-12-02T14:24:47Z" level=info msg="cleaning up pod" action=labelPodCompleted key=<namespace>/regression-test-2/labelPodCompleted namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:24:47.765Z" level=info msg="Queueing Succeeded workflow <namespace>/regression-test-2 for delete in 5m0s due to TTL"
time="2025-12-02T14:24:47Z" level=info msg="delete pod event" pod=regression-test-2
time="2025-12-02T14:24:57.773Z" level=debug msg="Won't process Workflow since it's completed" key=<namespace>/regression-test-2
time="2025-12-02T14:25:07Z" level=info msg="cleaning up pod" action=killContainers key=<namespace>/regression-test-2/killContainers namespace=<namespace> podName=regression-test-2
time="2025-12-02T14:29:48.001Z" level=info msg="Deleting garbage collected workflow '<namespace>/regression-test-2'"
time="2025-12-02T14:29:48.015Z" level=info msg="Successfully request '<namespace>/regression-test-2' to be deleted"


3.6.12 logs:

time="2025-12-02T14:28:36.771Z" level=info msg="Processing workflow" Phase= ResourceVersion=2089960120 namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.790Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=0 workflow=regression-test-24xsq
time="2025-12-02T14:28:36.790Z" level=info msg="Updated phase  -> Running" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.793Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.794Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.794Z" level=info msg="Pod node regression-test-24xsq initialized Pending" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.794Z" level=warning msg="couldn't get boundaryTemplate through nodeName regression-test-24xsq" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.986Z" level=info msg="add pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:36.986Z" level=info msg="Created pod: regression-test-24xsq (regression-test-24xsq)" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.986Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.986Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:36.998Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=2089960131 workflow=regression-test-24xsq
time="2025-12-02T14:28:36.998Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:37.027Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:38.473Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:39.467Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:40.330Z" level=info msg="insignificant pod change" key=<namespace>/regression-test-24xsq
time="2025-12-02T14:28:40.474Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:40.496Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:41.472Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:42.484Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:44.486Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:45.490Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:46.496Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:46.987Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2089960131 namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:46.987Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-24xsq
time="2025-12-02T14:28:46.988Z" level=info msg="task-result changed" namespace=<namespace> nodeID=regression-test-24xsq workflow=regression-test-24xsq
time="2025-12-02T14:28:46.988Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:28:46.988Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Running new.progress=0/1 nodeID=regression-test-24xsq old.message= old.phase=Pending old.progress=0/1 workflow=regression-test-24xsq
time="2025-12-02T14:28:46.988Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:46.988Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:28:46.993Z" level=info msg="cleaning up pod" action=terminateContainers key=<namespace>/regression-test-24xsq/terminateContainers namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:28:46.993Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:28:47.002Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Running resourceVersion=2089960459 workflow=regression-test-24xsq
time="2025-12-02T14:28:52.509Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:28:54.522Z" level=info msg="update pod event" pod=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2089960459 namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="Task-result reconciliation" namespace=<namespace> numObjs=1 workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="node changed" namespace=<namespace> new.message= new.phase=Succeeded new.progress=0/1 nodeID=regression-test-24xsq old.message= old.phase=Running old.progress=0/1 workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="TaskSet Reconciliation" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg=reconcileAgentPod namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="Updated phase Running -> Succeeded" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="Marking workflow completed" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.511Z" level=info msg="Marking workflow as pending archiving" namespace=<namespace> workflow=regression-test-24xsq
time="2025-12-02T14:29:02.522Z" level=info msg="Workflow update successful" namespace=<namespace> phase=Succeeded resourceVersion=2089960891 workflow=regression-test-24xsq
time="2025-12-02T14:29:02.540Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:29:02.540Z" level=info msg="archiving workflow" namespace=<namespace> uid=c09a34d6-76c2-4261-b3af-d801c0811e24 workflow=regression-test-24xsq
time="2025-12-02T14:29:02.546Z" level=info msg="cleaning up pod" action=labelPodCompleted key=<namespace>/regression-test-24xsq/labelPodCompleted namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:29:02.582Z" level=info msg="delete pod event" pod=regression-test-24xsq
time="2025-12-02T14:29:02.595Z" level=info msg="Queueing Succeeded workflow <namespace>/regression-test-24xsq for delete in 5m0s due to TTL"
time="2025-12-02T14:29:16.994Z" level=info msg="cleaning up pod" action=killContainers key=<namespace>/regression-test-24xsq/killContainers namespace=<namespace> podName=regression-test-24xsq
time="2025-12-02T14:32:50.396Z" level=info msg="Queueing Succeeded workflow <namespace>/regression-test-24xsq for delete in 1m12s due to TTL"
time="2025-12-02T14:34:03.000Z" level=info msg="Deleting garbage collected workflow '<namespace>/regression-test-24xsq'"
time="2025-12-02T14:34:03.016Z" level=info msg="Successfully request '<namespace>/regression-test-24xsq' to be deleted"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Dec 02 '25 14:12 dmarquez-splunk

Potentially same root cause as https://github.com/argoproj/argo-workflows/issues/15008, but for workflow.labels (i.e. referencing an unresolved value within a sprig expression), so would require the extra "workflow.labels" string here:

https://github.com/argoproj/argo-workflows/blob/cfdfea7a4789f4025e516dca9eb5b242cc573b4a/util/template/expression_template.go#L33-L34

Dec 03 '25 17:12 elliotgunton

@elliotgunton thanks for taking a look at this! Can we expect the fixed to be back ported to 3.6 or would this only be fixed in 3.7?

Dec 03 '25 20:12 dmarquez-splunk

@elliotgunton I've made the change and added an e2e test to validate its proper value gets rendered to no luck

Dec 08 '25 19:12 dmarquez-splunk

I've found the source of the issue:

Seems that the upgrade from github.com/expr-lang/expr v1.16.9 to github.com/expr-lang/expr v1.17.0 is causing this issue.

Once I revert back to 1.16.9 on 3.6.12 commit then the original functionality is restored

Dec 08 '25 21:12 dmarquez-splunk

Debug error being logged:

[controller] (running)  time=2025-12-08T21:43:39.098Z level=DEBUG msg="Result and error are unresolved" component=workflow_worker workflow=sprig-dig-regression-test-c47rq namespace=argo error="runtime error: index out of range [3] with length 1 (1:7)\n | sprig.dig('some', 'label', 'io/creator-email', nil, workflow.labels)\n | ......^"

Dec 08 '25 21:12 dmarquez-splunk

FYI https://github.com/expr-lang/expr/issues/875

Dec 09 '25 20:12 dmarquez-splunk

@Joibel do you think we can revert the version of expr-lang/expr to pre 1.17 version?

Dec 15 '25 22:12 dmarquez-splunk

@dmarquez-splunk Good job tracking this down! I entered https://github.com/argoproj/argo-workflows/pull/15168 to upgrade expr to a version with the fix, and included a test case.

Dec 16 '25 03:12 MasonM

@Joibel do you think we can revert the version of expr-lang/expr to pre 1.17 version?

We can't go backwards due to dependencies on argo-events and a CVE there (which is why we had to update in a patch release). Thanks to you and Mason we look to have a fix, so I'll try and coerce that through

Dec 16 '25 09:12 Joibel