DAG improperly executes child of conditional tasks that evaluate false

[gisjedi]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I can confirm the issues exists when I tested with :latest
• [ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I am using the DAG conditional logic to evaluate workflow tasks that should not be run. Take the following simple example case:

A -> B* -> C

A: DAG task that generates value to evaluate B*: DAG task that is conditional based on value from A and with depends: A C: DAG Task with depends: B

If B* conditional evaluates to false, I would expect both B* and C tasks to not execute. This is not the behavior Argo Workflow currently exhibits as of v3.4.0.

I've utilized the coin flip example and modified it to demonstrate this issue and it is included in workflow attached to issue.

As can be seen below, the submission of the above workflow executes both post-heads and post-tails. It should only execute one of these based on the coin flip.

Version

v3.4.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-conditional-artifacts-
  labels:
    workflows.argoproj.io/test: "true"
  annotations:
    workflows.argoproj.io/description: |
      Conditional artifacts provides a way to choose the output artifacts based on an expression.
      In this example the DAG template has two tasks which will run conditionall using `when`.
      Based on the condition one of steps may not execute. The step template output's artifact will be set to the
      executed step's output artifacts.
    workflows.argoproj.io/version: '>= 3.1.0'
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: flip-coin
            template: flip-coin
          - name: heads
            depends: flip-coin
            template: heads
            when: "{{tasks.flip-coin.outputs.result}} == heads"
          - name: post-heads
            depends: heads
            template: heads
          - name: tails
            depends: flip-coin
            template: tails
            when: "{{tasks.flip-coin.outputs.result}} == tails"
          - name: post-tails
            depends: tails
            template: tails
      outputs:
        artifacts:
          - name: result
            fromExpression: "tasks['flip-coin'].outputs.result == 'heads' ? tasks.heads.outputs.artifacts.result : tasks.tails.outputs.artifacts.result"

    - name: flip-coin
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          import random
          print("heads" if random.randint(0,1) == 0 else "tails")
    - name: heads
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          with open("result.txt", "w") as f:
            f.write("it was heads")
      outputs:
        artifacts:
          - name: result
            path: /result.txt

    - name: tails
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          with open("result.txt", "w") as f:
            f.write("it was tails")
      outputs:
        artifacts:
          - name: result
            path: /result.txt

Logs from the workflow controller

time="2022-09-19T21:49:04.820Z" level=info msg="Processing workflow" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.823Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.823Z" level=info msg="DAG node dag-conditional-artifacts-hsqzc initialized Running" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.824Z" level=info msg="All of node dag-conditional-artifacts-hsqzc.flip-coin dependencies [] completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.824Z" level=info msg="Pod node dag-conditional-artifacts-hsqzc-4073208811 initialized Pending" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.831Z" level=info msg="Created pod: dag-conditional-artifacts-hsqzc.flip-coin (dag-conditional-artifacts-hsqzc-flip-coin-4073208811)" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.831Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.831Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:04.839Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=23955 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.820Z" level=info msg="Processing workflow" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="task-result changed" namespace=argo nodeID=dag-conditional-artifacts-hsqzc-4073208811 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-conditional-artifacts-hsqzc-4073208811 old.message= old.phase=Pending old.progress=0/1 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="All of node dag-conditional-artifacts-hsqzc.heads dependencies [flip-coin] completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="Skipped node dag-conditional-artifacts-hsqzc-4102279299 initialized Skipped (message: when 'tails == heads' evaluated false)" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="All of node dag-conditional-artifacts-hsqzc.post-heads dependencies [heads] completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.821Z" level=info msg="Pod node dag-conditional-artifacts-hsqzc-1891345328 initialized Pending" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.826Z" level=info msg="Created pod: dag-conditional-artifacts-hsqzc.post-heads (dag-conditional-artifacts-hsqzc-heads-1891345328)" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.826Z" level=info msg="All of node dag-conditional-artifacts-hsqzc.tails dependencies [flip-coin] completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.826Z" level=info msg="Pod node dag-conditional-artifacts-hsqzc-4018639763 initialized Pending" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.831Z" level=info msg="Created pod: dag-conditional-artifacts-hsqzc.tails (dag-conditional-artifacts-hsqzc-tails-4018639763)" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.831Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.831Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.840Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=24010 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:14.846Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-conditional-artifacts-hsqzc-flip-coin-4073208811/labelPodCompleted
time="2022-09-19T21:49:24.831Z" level=info msg="Processing workflow" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="task-result changed" namespace=argo nodeID=dag-conditional-artifacts-hsqzc-1891345328 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="task-result changed" namespace=argo nodeID=dag-conditional-artifacts-hsqzc-4018639763 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-conditional-artifacts-hsqzc-4018639763 old.message= old.phase=Pending old.progress=0/1 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-conditional-artifacts-hsqzc-1891345328 old.message= old.phase=Pending old.progress=0/1 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="All of node dag-conditional-artifacts-hsqzc.post-tails dependencies [tails] completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.831Z" level=info msg="Pod node dag-conditional-artifacts-hsqzc-1358398416 initialized Pending" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.836Z" level=info msg="Created pod: dag-conditional-artifacts-hsqzc.post-tails (dag-conditional-artifacts-hsqzc-tails-1358398416)" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.836Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.836Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.844Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=24086 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:24.849Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-conditional-artifacts-hsqzc-heads-1891345328/labelPodCompleted
time="2022-09-19T21:49:24.849Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-conditional-artifacts-hsqzc-tails-4018639763/labelPodCompleted
time="2022-09-19T21:49:34.838Z" level=info msg="Processing workflow" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.838Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.838Z" level=info msg="task-result changed" namespace=argo nodeID=dag-conditional-artifacts-hsqzc-1358398416 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.838Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-conditional-artifacts-hsqzc-1358398416 old.message= old.phase=Pending old.progress=0/1 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Outbound nodes of dag-conditional-artifacts-hsqzc set to [dag-conditional-artifacts-hsqzc-1891345328 dag-conditional-artifacts-hsqzc-1358398416]" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="node dag-conditional-artifacts-hsqzc phase Running -> Succeeded" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="node dag-conditional-artifacts-hsqzc finished: 2022-09-19 21:49:34.840136713 +0000 UTC" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Checking daemoned children of dag-conditional-artifacts-hsqzc" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Marking workflow completed" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.840Z" level=info msg="Checking daemoned children of " namespace=argo workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.845Z" level=info msg="cleaning up pod" action=deletePod key=argo/dag-conditional-artifacts-hsqzc-1340600742-agent/deletePod
time="2022-09-19T21:49:34.846Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=24145 workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.861Z" level=info msg="archiving workflow" namespace=argo uid=ce77c689-b215-4a3c-8f86-7c7849aa037e workflow=dag-conditional-artifacts-hsqzc
time="2022-09-19T21:49:34.866Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-conditional-artifacts-hsqzc-tails-1358398416/labelPodCompleted

Logs from in your workflow's wait container

time="2022-09-19T21:49:06.356Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=true namespace=argo podName=dag-conditional-artifacts-hsqzc-flip-coin-4073208811 template="{\"name\":\"flip-coin\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"script\":{\"name\":\"\",\"image\":\"python:alpine3.6\",\"command\":[\"python\"],\"resources\":{},\"source\":\"import random\\nprint(\\\"heads\\\" if random.randint(0,1) == 0 else \\\"tails\\\")\\n\"}}" version="&Version{Version:v3.4.0-rc3,BuildDate:2022-09-01T00:17:58Z,GitCommit:1696fb21454914552e3451bc6eb9a780b2eec10b,GitTag:v3.4.0-rc3,GitTreeState:clean,GoVersion:go1.18.5,Compiler:gc,Platform:linux/arm64,}"
time="2022-09-19T21:49:06.356Z" level=info msg="Starting deadline monitor"
time="2022-09-19T21:49:08.357Z" level=info msg="Main container completed" error="<nil>"
time="2022-09-19T21:49:08.357Z" level=info msg="Capturing script output"
time="2022-09-19T21:49:08.357Z" level=info msg="No output parameters"
time="2022-09-19T21:49:08.357Z" level=info msg="No output artifacts"
time="2022-09-19T21:49:08.364Z" level=info msg="Create workflowtaskresults 201"
time="2022-09-19T21:49:08.365Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-09-19T21:49:08.365Z" level=info msg="Deadline monitor stopped"
time="2022-09-19T21:49:08.365Z" level=info msg="Alloc=6096 TotalAlloc=12246 Sys=18898 NumGC=4 Goroutines=5"
time="2022-09-19T21:49:18.612Z" level=info msg="/var/run/argo/outputs/artifacts/result.txt.tgz -> /tmp/argo/outputs/artifacts/result.tgz"
time="2022-09-19T21:49:18.612Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/result.tgz, key: dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-heads-1891345328/result.tgz"
time="2022-09-19T21:49:18.612Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2022-09-19T21:49:18.612Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-heads-1891345328/result.tgz path=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:18.617Z" level=info msg="Save artifact" artifactName=result duration=5.174167ms error="<nil>" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-heads-1891345328/result.tgz
time="2022-09-19T21:49:18.617Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:18.617Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/result.tgz"
time="2022-09-19T21:49:18.623Z" level=info msg="Create workflowtaskresults 201"
time="2022-09-19T21:49:18.623Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-09-19T21:49:18.624Z" level=info msg="Alloc=7264 TotalAlloc=12407 Sys=18642 NumGC=3 Goroutines=9"
time="2022-09-19T21:49:27.917Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/result.tgz, key: dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-1358398416/result.tgz"
time="2022-09-19T21:49:27.917Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2022-09-19T21:49:27.917Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-1358398416/result.tgz path=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:27.922Z" level=info msg="Save artifact" artifactName=result duration=4.981625ms error="<nil>" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-1358398416/result.tgz
time="2022-09-19T21:49:27.922Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:27.922Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/result.tgz"
time="2022-09-19T21:49:27.928Z" level=info msg="Create workflowtaskresults 201"
time="2022-09-19T21:49:27.928Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-09-19T21:49:27.928Z" level=info msg="Deadline monitor stopped"
time="2022-09-19T21:49:27.929Z" level=info msg="Alloc=6783 TotalAlloc=12643 Sys=18642 NumGC=4 Goroutines=7"
time="2022-09-19T21:49:18.634Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/result.tgz, key: dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-4018639763/result.tgz"
time="2022-09-19T21:49:18.634Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2022-09-19T21:49:18.634Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-4018639763/result.tgz path=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:18.641Z" level=info msg="Save artifact" artifactName=result duration=6.948458ms error="<nil>" key=dag-conditional-artifacts-hsqzc/dag-conditional-artifacts-hsqzc-tails-4018639763/result.tgz
time="2022-09-19T21:49:18.641Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/result.tgz
time="2022-09-19T21:49:18.641Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/result.tgz"
time="2022-09-19T21:49:18.648Z" level=info msg="Create workflowtaskresults 201"
time="2022-09-19T21:49:18.649Z" level=info msg="Deadline monitor stopped"
time="2022-09-19T21:49:18.649Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-09-19T21:49:18.649Z" level=info msg="Alloc=6809 TotalAlloc=12570 Sys=18642 NumGC=4 Goroutines=9"

[gisjedi]

This is a priority for our team and I'm happy to help with resolving this if support is needed.

[tooptoop4]

does it work in 3.3.9?

[gisjedi]

I've tested on 3.2.6, 3.3.9 and 3.4.0. They all have the same behavior.

[sarabala1979]

@gisjedi Do you like to fix this issue? Please go ahead to submit the PR. We are happy to review it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.