argo-workflows icon indicating copy to clipboard operation
argo-workflows copied to clipboard

Published 20 hours ago • verified publisher icon argoproj

Multiple references to consecutive withParam in same workflow will fail.

Open AlanD88 opened this issue 1 year ago • 1 comments

Checklist

  • [x] Double-checked my configuration.
  • [x] Tested using the latest version.
  • [x] Used the Emissary executor.

Summary

When using withParam to read values from a json file written by previous steps, the workflow will fail to parse the json file upon the 3rd consecutive reference to withParam.

To emphasize the issue, i have written a test workflow with the 2nd task calling withParam (Task C) but not actually relying on any values from it. This will cause the subsequent task (Task D) to fail at parsing its json input, but if withParam is commented out in task C the subsequent task D will parse the json input just fine.

What version are you running? v3.3.9

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-diamond-jsontest-
  annotations:
    workflows.argoproj.io/description: |
      This is an example of of breaking withParam.
spec:
  serviceAccountName: workflow
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: A
        template: writeJson
        arguments:
          parameters:
            - name: dummyv1
              value: "dv1"
            - name: dummyv2
              value: "dv2"
      - name: B
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.A.outputs.parameters.stepout}}"
        depends: "A"
        template: writeJson
      - name: C
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.B.outputs.parameters.stepout}}"
        depends: "B"
        template: writeJson
      - name: D
        arguments:
          parameters:
            - name: dummyv1
              value: "{{item.dummyv1}}"
            - name: dummyv2
              value: "{{item.dummyv2}}"
        withParam: "{{tasks.C.outputs.parameters.stepout}}"
        depends: "B && C"
        template: writeJson


  - name: writeJson
    inputs:
      parameters:
      - name: dummyv1
      - name: dummyv2
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import json
        args = [dict(dummyv1='one', dummyv2='two')]
        with open("/tmp/myfile.json", "w") as fd:
          tmp = json.dumps(args)
          fd.write(tmp)
    outputs:
      parameters:
      - name: stepout
        valueFrom: 
          path: /tmp/myfile.json

STEP                                                      TEMPLATE   PODNAME                                DURATION  MESSAGE
 ⚠ dag-diamond-jsontest-kf7td                             diamond                                                                                         
 ├─✔ A                                                    writeJson  dag-diamond-jsontest-kf7td-1806111301  3s                                            
 ├─✔ B(0:dummyv1:one,dummyv2:two)                         writeJson  dag-diamond-jsontest-kf7td-3988660372  3s                                            
 ├─✔ C(0:[{\"dummyv1\": \"one\", \"dummyv2\": \"two\"}])  writeJson  dag-diamond-jsontest-kf7td-913068289   3s                                            
 └─⚠ D                                                    writeJson                                                   failed to resolve {{item.dummyv1}}

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow} 
time="2022-08-30T23:35:01.305Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Updated phase  -> Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="DAG node dag-diamond-jsontest-kf7td initialized Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="All of node dag-diamond-jsontest-kf7td.A dependencies [] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-1806111301 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.A (dag-diamond-jsontest-kf7td-1806111301)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.318Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6712 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-1806111301 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:11.318Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1755778444 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="All of node dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) dependencies [A] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-3988660372 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) (dag-diamond-jsontest-kf7td-3988660372)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.330Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6744 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.336Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-1806111301/labelPodCompleted
time="2022-08-30T23:35:21.329Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.330Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-3988660372 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 finished: 2022-08-30 23:35:21.33152398 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1772556063 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="All of node dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) dependencies [B] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-913068289 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) (dag-diamond-jsontest-kf7td-913068289)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.344Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6782 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.350Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-3988660372/labelPodCompleted
time="2022-08-30T23:35:31.343Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-913068289 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 finished: 2022-08-30 23:35:31.345323089 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="Skipped node dag-diamond-jsontest-kf7td-1722223206 initialized Error (message: failed to resolve {{item.dummyv2}})" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Outbound nodes of dag-diamond-jsontest-kf7td set to [dag-diamond-jsontest-kf7td-1722223206]" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td finished: 2022-08-30 23:35:31.346203587 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of dag-diamond-jsontest-kf7td" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Marking workflow completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of " namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.352Z" level=info msg="cleaning up pod" action=deletePod key=default/dag-diamond-jsontest-kf7td-1340600742-agent/deletePod
time="2022-08-30T23:35:31.381Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=6821 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.393Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-913068289/labelPodCompleted


# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Here is an example i meant for how commenting out one of the withParams for task C will now permit this workflow to complete for task D. Even though it should have changed nothing for task D as far as I understand it

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-diamond-jsontest-
  annotations:
    workflows.argoproj.io/description: |
      This is an example of of breaking withParam.
spec:
  serviceAccountName: workflow
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: A
        template: writeJson
        arguments:
          parameters:
            - name: dummyv1
              value: "dv1"
            - name: dummyv2
              value: "dv2"
      - name: B
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.A.outputs.parameters.stepout}}"
        depends: "A"
        template: writeJson
      - name: C
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
#        withParam: "{{tasks.B.outputs.parameters.stepout}}"
        depends: "B"
        template: writeJson
      - name: D
        arguments:
          parameters:
            - name: dummyv1
              value: "{{item.dummyv1}}"
            - name: dummyv2
              value: "{{item.dummyv2}}"
        withParam: "{{tasks.C.outputs.parameters.stepout}}"
        depends: "B && C"
        template: writeJson


  - name: writeJson
    inputs:
      parameters:
      - name: dummyv1
      - name: dummyv2
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import json
        args = [dict(dummyv1='one', dummyv2='two')]
        with open("/tmp/myfile.json", "w") as fd:
          tmp = json.dumps(args)
          fd.write(tmp)
    outputs:
      parameters:
      - name: stepout
        valueFrom: 
          path: /tmp/myfile.json

STEP                               TEMPLATE   PODNAME                                DURATION  MESSAGE
 ✔ dag-diamond-jsontest-nklct      diamond                                                       
 ├─✔ A                             writeJson  dag-diamond-jsontest-nklct-3272336441  4s          
 ├─✔ B(0:dummyv1:one,dummyv2:two)  writeJson  dag-diamond-jsontest-nklct-2037240488  4s          
 ├─✔ C                             writeJson  dag-diamond-jsontest-nklct-3238781203  4s          
 └─✔ D(0:dummyv1:one,dummyv2:two)  writeJson  dag-diamond-jsontest-nklct-3224523118  4s

As far as I can tell reading the docs I never came across a section that seemed to imply there was a limitation on how many times you could reference using a json file as input and reading it via withParams within a single workflow

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

AlanD88 avatar Aug 30 '22 23:08 AlanD88

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

stale[bot] avatar Oct 01 '22 17:10 stale[bot]

I had to slightly change your example to fix some other bugs that came up. But I was able to get this to work on 6b086368f6480a2de5e2d43eec73514de0ad01ac. It fails to make any progress at all on v3.3.9 though. @AlanD88 would you be able to please confirm if the following works for you or not? Thanks

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-diamond-jsontest-
  annotations:
    workflows.argoproj.io/description: |
      This is an example of of breaking withParam.
spec:
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: a
        template: write-json
        arguments:
          parameters:
            - name: dummyv1
              value: "dv1"
            - name: dummyv2
              value: "dv2"
      - name: b
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.a.outputs.parameters.stepout}}"
        depends: "a"
        template: write-json
      - name: c
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.b.outputs.parameters.stepout}}"
        depends: "b"
        template: write-json
      - name: d
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.c.outputs.parameters.stepout}}"
        depends: "b && c"
        template: write-json


  - name: write-json
    inputs:
      parameters:
      - name: dummyv1
      - name: dummyv2
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import json
        args = [dict(dummyv1='one', dummyv2='two')]
        with open("/tmp/myfile.json", "w") as fd:
          tmp = json.dumps(args)
          fd.write(tmp)
    outputs:
      parameters:
      - name: stepout
        valueFrom: 
          path: /tmp/myfile.json

isubasinghe avatar Oct 25 '22 05:10 isubasinghe

v3.4.3 was released yesterday. @AlanD88 are you able to test and confirm on latest for us? Thanks

caelan-io avatar Oct 31 '22 18:10 caelan-io

@caelan-io sorry things have been a bit hectic for me, i'll get a chance to test this out this week and report back.

AlanD88 avatar Nov 02 '22 07:11 AlanD88

@isubasinghe @caelan-io Sorry it took a bit, was able to test locally. Looks to work fine now.

AlanD88 avatar Nov 09 '22 08:11 AlanD88

Fantastic! All good, and thanks a lot for confirming for us, @AlanD88. That's really helpful. We'll close this issue as fixed in v3.4.3.

caelan-io avatar Nov 09 '22 15:11 caelan-io

@caelan-io This needs to be reopened. I work in an air gapped environment, so when I tested this, i started up a session on my home computer and tested and just copied @isubasinghe template to do the quick test again. When my internal system at work got updated v3.4.3 and I attempted to retest our workflow design I noticed this issue was actually still occurring. Looking back through the template @isubasinghe modified, I noticed they actually removed the logic from step D that was demonstrating the issue where on the 3rd attempt reading dynamic input, it will no longer correctly parse a json input artifact so you can't do key access on the contents like you normally should be able to.

The point of step D was to demonstrate key access failure from the contents of the json, where normally it should succeed.

The step from my original template that will break

      - name: D
        arguments:
          parameters:
            - name: dummyv1
              value: "{{item.dummyv1}}"
            - name: dummyv2
              value: "{{item.dummyv2}}"
        withParam: "{{tasks.C.outputs.parameters.stepout}}"
        depends: "B && C"
        template: writeJson

Where as the template that was tested to demonstrate it had been fixed

      - name: d
        arguments:
          parameters:
            - name: dummyv1
              value: "tmp"
            - name: dummyv2
              value: "tmp2"
            - name: jsonOut
              value: "{{item}}"
        withParam: "{{tasks.c.outputs.parameters.stepout}}"
        depends: "b && c"
        template: write-json

AlanD88 avatar Nov 20 '22 03:11 AlanD88

Thanks for checking on this for me @AlanD88, I will re-open this in that case. I can confirm that actually using the values from step c does cause the issue to reappear. I will have a look at this tomorrow.

isubasinghe avatar Nov 20 '22 04:11 isubasinghe

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

stale[bot] avatar Dec 31 '22 22:12 stale[bot]

https://github.com/isubasinghe/argo-workflows/blob/multiple-references-to-consecutive/output.txt

isubasinghe avatar Feb 24 '23 01:02 isubasinghe

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

stale[bot] avatar Mar 25 '23 06:03 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale[bot] avatar Sep 17 '23 21:09 stale[bot]

Any update on this issue?

idojustt avatar Feb 19 '24 18:02 idojustt