argo-workflows
argo-workflows copied to clipboard
Multiple references to consecutive withParam in same workflow will fail.
Checklist
- [x] Double-checked my configuration.
- [x] Tested using the latest version.
- [x] Used the Emissary executor.
Summary
When using withParam to read values from a json file written by previous steps, the workflow will fail to parse the json file upon the 3rd consecutive reference to withParam.
To emphasize the issue, i have written a test workflow with the 2nd task calling withParam (Task C) but not actually relying on any values from it. This will cause the subsequent task (Task D) to fail at parsing its json input, but if withParam is commented out in task C the subsequent task D will parse the json input just fine.
What version are you running? v3.3.9
Diagnostics
Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-jsontest-
annotations:
workflows.argoproj.io/description: |
This is an example of of breaking withParam.
spec:
serviceAccountName: workflow
entrypoint: diamond
templates:
- name: diamond
dag:
tasks:
- name: A
template: writeJson
arguments:
parameters:
- name: dummyv1
value: "dv1"
- name: dummyv2
value: "dv2"
- name: B
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.A.outputs.parameters.stepout}}"
depends: "A"
template: writeJson
- name: C
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.B.outputs.parameters.stepout}}"
depends: "B"
template: writeJson
- name: D
arguments:
parameters:
- name: dummyv1
value: "{{item.dummyv1}}"
- name: dummyv2
value: "{{item.dummyv2}}"
withParam: "{{tasks.C.outputs.parameters.stepout}}"
depends: "B && C"
template: writeJson
- name: writeJson
inputs:
parameters:
- name: dummyv1
- name: dummyv2
script:
image: python:alpine3.6
command: [python]
source: |
import json
args = [dict(dummyv1='one', dummyv2='two')]
with open("/tmp/myfile.json", "w") as fd:
tmp = json.dumps(args)
fd.write(tmp)
outputs:
parameters:
- name: stepout
valueFrom:
path: /tmp/myfile.json
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ dag-diamond-jsontest-kf7td diamond
├─✔ A writeJson dag-diamond-jsontest-kf7td-1806111301 3s
├─✔ B(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-kf7td-3988660372 3s
├─✔ C(0:[{\"dummyv1\": \"one\", \"dummyv2\": \"two\"}]) writeJson dag-diamond-jsontest-kf7td-913068289 3s
└─⚠ D writeJson failed to resolve {{item.dummyv1}}
# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2022-08-30T23:35:01.305Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Updated phase -> Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="DAG node dag-diamond-jsontest-kf7td initialized Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="All of node dag-diamond-jsontest-kf7td.A dependencies [] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-1806111301 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.A (dag-diamond-jsontest-kf7td-1806111301)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.318Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6712 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-1806111301 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:11.318Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1755778444 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="All of node dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) dependencies [A] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-3988660372 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) (dag-diamond-jsontest-kf7td-3988660372)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.330Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6744 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.336Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-1806111301/labelPodCompleted
time="2022-08-30T23:35:21.329Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.330Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-3988660372 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 finished: 2022-08-30 23:35:21.33152398 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1772556063 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="All of node dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) dependencies [B] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-913068289 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) (dag-diamond-jsontest-kf7td-913068289)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.344Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6782 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.350Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-3988660372/labelPodCompleted
time="2022-08-30T23:35:31.343Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-913068289 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 finished: 2022-08-30 23:35:31.345323089 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="Skipped node dag-diamond-jsontest-kf7td-1722223206 initialized Error (message: failed to resolve {{item.dummyv2}})" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Outbound nodes of dag-diamond-jsontest-kf7td set to [dag-diamond-jsontest-kf7td-1722223206]" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td finished: 2022-08-30 23:35:31.346203587 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of dag-diamond-jsontest-kf7td" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Marking workflow completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of " namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.352Z" level=info msg="cleaning up pod" action=deletePod key=default/dag-diamond-jsontest-kf7td-1340600742-agent/deletePod
time="2022-08-30T23:35:31.381Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=6821 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.393Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-913068289/labelPodCompleted
# If the workflow's pods have not been created, you can skip the rest of the diagnostics.
# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
Here is an example i meant for how commenting out one of the withParams for task C will now permit this workflow to complete for task D. Even though it should have changed nothing for task D as far as I understand it
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-jsontest-
annotations:
workflows.argoproj.io/description: |
This is an example of of breaking withParam.
spec:
serviceAccountName: workflow
entrypoint: diamond
templates:
- name: diamond
dag:
tasks:
- name: A
template: writeJson
arguments:
parameters:
- name: dummyv1
value: "dv1"
- name: dummyv2
value: "dv2"
- name: B
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.A.outputs.parameters.stepout}}"
depends: "A"
template: writeJson
- name: C
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
# withParam: "{{tasks.B.outputs.parameters.stepout}}"
depends: "B"
template: writeJson
- name: D
arguments:
parameters:
- name: dummyv1
value: "{{item.dummyv1}}"
- name: dummyv2
value: "{{item.dummyv2}}"
withParam: "{{tasks.C.outputs.parameters.stepout}}"
depends: "B && C"
template: writeJson
- name: writeJson
inputs:
parameters:
- name: dummyv1
- name: dummyv2
script:
image: python:alpine3.6
command: [python]
source: |
import json
args = [dict(dummyv1='one', dummyv2='two')]
with open("/tmp/myfile.json", "w") as fd:
tmp = json.dumps(args)
fd.write(tmp)
outputs:
parameters:
- name: stepout
valueFrom:
path: /tmp/myfile.json
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ dag-diamond-jsontest-nklct diamond
├─✔ A writeJson dag-diamond-jsontest-nklct-3272336441 4s
├─✔ B(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-nklct-2037240488 4s
├─✔ C writeJson dag-diamond-jsontest-nklct-3238781203 4s
└─✔ D(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-nklct-3224523118 4s
As far as I can tell reading the docs I never came across a section that seemed to imply there was a limitation on how many times you could reference using a json file as input and reading it via withParams within a single workflow
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
I had to slightly change your example to fix some other bugs that came up. But I was able to get this to work on 6b086368f6480a2de5e2d43eec73514de0ad01ac. It fails to make any progress at all on v3.3.9 though. @AlanD88 would you be able to please confirm if the following works for you or not? Thanks
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-jsontest-
annotations:
workflows.argoproj.io/description: |
This is an example of of breaking withParam.
spec:
entrypoint: diamond
templates:
- name: diamond
dag:
tasks:
- name: a
template: write-json
arguments:
parameters:
- name: dummyv1
value: "dv1"
- name: dummyv2
value: "dv2"
- name: b
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.a.outputs.parameters.stepout}}"
depends: "a"
template: write-json
- name: c
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.b.outputs.parameters.stepout}}"
depends: "b"
template: write-json
- name: d
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.c.outputs.parameters.stepout}}"
depends: "b && c"
template: write-json
- name: write-json
inputs:
parameters:
- name: dummyv1
- name: dummyv2
script:
image: python:alpine3.6
command: [python]
source: |
import json
args = [dict(dummyv1='one', dummyv2='two')]
with open("/tmp/myfile.json", "w") as fd:
tmp = json.dumps(args)
fd.write(tmp)
outputs:
parameters:
- name: stepout
valueFrom:
path: /tmp/myfile.json
v3.4.3 was released yesterday. @AlanD88 are you able to test and confirm on latest
for us? Thanks
@caelan-io sorry things have been a bit hectic for me, i'll get a chance to test this out this week and report back.
@isubasinghe @caelan-io Sorry it took a bit, was able to test locally. Looks to work fine now.
Fantastic! All good, and thanks a lot for confirming for us, @AlanD88. That's really helpful. We'll close this issue as fixed in v3.4.3.
@caelan-io This needs to be reopened. I work in an air gapped environment, so when I tested this, i started up a session on my home computer and tested and just copied @isubasinghe template to do the quick test again. When my internal system at work got updated v3.4.3 and I attempted to retest our workflow design I noticed this issue was actually still occurring. Looking back through the template @isubasinghe modified, I noticed they actually removed the logic from step D that was demonstrating the issue where on the 3rd attempt reading dynamic input, it will no longer correctly parse a json input artifact so you can't do key access on the contents like you normally should be able to.
The point of step D was to demonstrate key access failure from the contents of the json, where normally it should succeed.
The step from my original template that will break
- name: D
arguments:
parameters:
- name: dummyv1
value: "{{item.dummyv1}}"
- name: dummyv2
value: "{{item.dummyv2}}"
withParam: "{{tasks.C.outputs.parameters.stepout}}"
depends: "B && C"
template: writeJson
Where as the template that was tested to demonstrate it had been fixed
- name: d
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.c.outputs.parameters.stepout}}"
depends: "b && c"
template: write-json
Thanks for checking on this for me @AlanD88, I will re-open this in that case. I can confirm that actually using the values from step c does cause the issue to reappear. I will have a look at this tomorrow.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
https://github.com/isubasinghe/argo-workflows/blob/multiple-references-to-consecutive/output.txt
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Any update on this issue?