argo-workflows Retry does not work correctly with a DAG when there are steps which have dependencies

Retry does not work correctly with a DAG when there are steps which have dependencies

Open RoryDoherty opened this issue 2 years ago • 12 comments

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issues exists when I tested with :latest
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

If you have an argo workflow dag with the following:

flowchart TD
A[A - Setup test infrastructure] --> B[ B - Run Tests 1]
A --> C[C - Run Tests 2]
A --> D[D - Run Tests 2]
A --> E[E - Run Tests 2]
B --> F[OnExit - Cleanup test Infrastructure]
C --> F
D --> F
E --> F

In the event of everything passing except for step C I may want to only retry this step. However if I click Retry on the UI then it will only re run the C step which will obviously fail again as the infrastructure does not exist that step A sets up even though in the dag I have specified dependsOn A Also the OnExit steps will not run again in the event of a retry, is there a way to specify that these should always run even during a retry? More info can be found here https://github.com/argoproj/argo-workflows/discussions/7534

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: demo-
spec:
  entrypoint: demo
  onExit: cleanup
  templates:
    - name: demo
      dag:
        tasks:
          - name: a
            template: setup
          - name: b
            depends: a
            template: run-test
          - name: c
            depends: a
            template: fail-test
          - name: d
            depends: a
            template: run-test
          - name: e
            depends: a
            template: run-test

    - name: setup
      script:
        image: ubuntu:18.04
        imagePullPolicy: Always
        command: [bash]
        workingDir: "/src"
        source: |
          sleep 5
          echo Running setup step
          date
          exit 0

    - name: run-test
      script:
        image: ubuntu:18.04
        imagePullPolicy: Always
        command: [bash]
        workingDir: "/src"
        source: |
          sleep 5
          echo Running test success
          date
          exit 0

    - name: fail-test
      script:
        image: ubuntu:18.04
        imagePullPolicy: Always
        command: [bash]
        workingDir: "/src"
        source: |
          sleep 5
          echo Running test failure
          date
          exit 1

    - name: cleanup
      script:
        image: ubuntu:18.04
        imagePullPolicy: Always
        command: [bash]
        workingDir: "/src"
        source: |
          sleep 5
          echo Running cleanup step
          date
          exit 0

Logs from the workflow controller

time="2023-01-30T10:26:49.919Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.926Z" level=info msg="Updated phase  -> Running" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.926Z" level=info msg="DAG node demo-hsqz4 initialized Running" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.926Z" level=info msg="All of node demo-hsqz4.a dependencies [] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.926Z" level=info msg="Pod node demo-hsqz4-1639301744 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.946Z" level=info msg="Created pod: demo-hsqz4.a (demo-hsqz4-setup-1639301744)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.946Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.946Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:49.960Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862605919 workflow=demo-hsqz4
time="2023-01-30T10:26:59.947Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:59.948Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=0 workflow=demo-hsqz4
time="2023-01-30T10:26:59.948Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1639301744 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:26:59.949Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:59.949Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:26:59.954Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-setup-1639301744/terminateContainers
time="2023-01-30T10:26:59.956Z" level=info msg="https://10.164.16.1:443/api/v1/namespaces/argoci/pods/demo-hsqz4-setup-1639301744/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-01-30T10:26:59.963Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862606089 workflow=demo-hsqz4
time="2023-01-30T10:27:00.239Z" level=info msg="signaled container" container=wait error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"3ecd4703ab024b084c15d9cb94eb4e8eec1254ba6fa2363c6968417757b157ba\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown" namespace=argoci pod=demo-hsqz4-setup-1639301744 stderr="<nil>" stdout="<nil>"
time="2023-01-30T10:27:09.964Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.965Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=1 workflow=demo-hsqz4
time="2023-01-30T10:27:09.965Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1639301744 workflow=demo-hsqz4
time="2023-01-30T10:27:09.965Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-hsqz4-1639301744 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:09.965Z" level=info msg="All of node demo-hsqz4.b dependencies [a] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.965Z" level=info msg="Pod node demo-hsqz4-1689634601 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.979Z" level=info msg="Created pod: demo-hsqz4.b (demo-hsqz4-run-test-1689634601)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.980Z" level=info msg="All of node demo-hsqz4.c dependencies [a] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.980Z" level=info msg="Pod node demo-hsqz4-1672856982 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.994Z" level=info msg="Created pod: demo-hsqz4.c (demo-hsqz4-fail-test-1672856982)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.995Z" level=info msg="All of node demo-hsqz4.d dependencies [a] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:09.995Z" level=info msg="Pod node demo-hsqz4-1723189839 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.008Z" level=info msg="Created pod: demo-hsqz4.d (demo-hsqz4-run-test-1723189839)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.008Z" level=info msg="All of node demo-hsqz4.e dependencies [a] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.008Z" level=info msg="Pod node demo-hsqz4-1706412220 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.024Z" level=info msg="Created pod: demo-hsqz4.e (demo-hsqz4-run-test-1706412220)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.024Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.024Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:10.043Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862606251 workflow=demo-hsqz4
time="2023-01-30T10:27:15.045Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-setup-1639301744/deletePod
time="2023-01-30T10:27:19.981Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:19.982Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=2 workflow=demo-hsqz4
time="2023-01-30T10:27:19.983Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1672856982 workflow=demo-hsqz4
time="2023-01-30T10:27:19.983Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1706412220 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:19.983Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1723189839 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:19.983Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1672856982 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:19.983Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1689634601 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:19.984Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:19.984Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:19.988Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-run-test-1706412220/terminateContainers
time="2023-01-30T10:27:19.988Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-run-test-1723189839/terminateContainers
time="2023-01-30T10:27:19.988Z" level=info msg="https://10.164.16.1:443/api/v1/namespaces/argoci/pods/demo-hsqz4-run-test-1706412220/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-01-30T10:27:19.989Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-run-test-1689634601/terminateContainers
time="2023-01-30T10:27:19.989Z" level=info msg="https://10.164.16.1:443/api/v1/namespaces/argoci/pods/demo-hsqz4-run-test-1723189839/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-01-30T10:27:19.988Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-fail-test-1672856982/terminateContainers
time="2023-01-30T10:27:19.992Z" level=info msg="https://10.164.16.1:443/api/v1/namespaces/argoci/pods/demo-hsqz4-run-test-1689634601/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-01-30T10:27:19.997Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862606435 workflow=demo-hsqz4
time="2023-01-30T10:27:20.141Z" level=info msg="signaled container" container=wait error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"1eed666386f2d2cd6a2213706483922cb6b4aa3412e9ac74c660eb80d85b6ba3\": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown" namespace=argoci pod=demo-hsqz4-run-test-1689634601 stderr="<nil>" stdout="<nil>"
time="2023-01-30T10:27:20.167Z" level=info msg="signaled container" container=wait error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"ac9f0a0d61ac99d727d135a3bd718a88f6d027749e653a20bef29cdcc9bf9f86\": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer: unknown" namespace=argoci pod=demo-hsqz4-run-test-1706412220 stderr="<nil>" stdout="<nil>"
time="2023-01-30T10:27:20.179Z" level=info msg="signaled container" container=wait error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"f37a06b6328784cfdf863f338ad96bbbf29f57d2a5abeecce960045342cbe6a8\": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: process_linux.go:130: executing setns process caused: exit status 1: unknown" namespace=argoci pod=demo-hsqz4-run-test-1723189839 stderr="<nil>" stdout="<nil>"
time="2023-01-30T10:27:29.997Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=5 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1723189839 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1706412220 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1689634601 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-hsqz4-1706412220 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-hsqz4-1723189839 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-hsqz4-1689634601 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:29.999Z" level=info msg="node changed" namespace=argoci new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=demo-hsqz4-1672856982 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:30.000Z" level=info msg="Outbound nodes of demo-hsqz4 set to [demo-hsqz4-1689634601 demo-hsqz4-1672856982 demo-hsqz4-1723189839 demo-hsqz4-1706412220]" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.000Z" level=info msg="node demo-hsqz4 phase Running -> Failed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.000Z" level=info msg="node demo-hsqz4 finished: 2023-01-30 10:27:30.000828103 +0000 UTC" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.000Z" level=info msg="Checking daemoned children of demo-hsqz4" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.000Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.001Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.001Z" level=info msg="Running OnExit handler: cleanup" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.001Z" level=info msg="Pod node demo-hsqz4-1813653132 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.022Z" level=info msg="Created pod: demo-hsqz4.onExit (demo-hsqz4-cleanup-1813653132)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:30.036Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862606609 workflow=demo-hsqz4
time="2023-01-30T10:27:30.239Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-setup-1639301744/killContainers
time="2023-01-30T10:27:35.038Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-fail-test-1672856982/deletePod
time="2023-01-30T10:27:35.038Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-run-test-1689634601/deletePod
time="2023-01-30T10:27:35.038Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-run-test-1723189839/deletePod
time="2023-01-30T10:27:35.038Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-run-test-1706412220/deletePod
time="2023-01-30T10:27:40.023Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:40.024Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=6 workflow=demo-hsqz4
time="2023-01-30T10:27:40.025Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1813653132 workflow=demo-hsqz4
time="2023-01-30T10:27:40.025Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1813653132 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:40.025Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:40.025Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:40.025Z" level=info msg="Running OnExit handler: cleanup" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:40.030Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-cleanup-1813653132/terminateContainers
time="2023-01-30T10:27:40.038Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862606761 workflow=demo-hsqz4
time="2023-01-30T10:27:49.990Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-fail-test-1672856982/killContainers
time="2023-01-30T10:27:50.040Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=6 workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-hsqz4-1813653132 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Running OnExit handler: cleanup" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Updated phase Running -> Failed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Marking workflow completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Marking workflow as pending archiving" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.042Z" level=info msg="Checking daemoned children of " namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:27:50.048Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-1340600742-agent/deletePod
time="2023-01-30T10:27:50.058Z" level=info msg="Workflow update successful" namespace=argoci phase=Failed resourceVersion=862606889 workflow=demo-hsqz4
time="2023-01-30T10:27:50.142Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-run-test-1689634601/killContainers
time="2023-01-30T10:27:50.145Z" level=info msg="archiving workflow" namespace=argoci uid=3719303d-fee6-4419-8d06-a068474027e9 workflow=demo-hsqz4
time="2023-01-30T10:27:50.167Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-run-test-1706412220/killContainers
time="2023-01-30T10:27:50.180Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-run-test-1723189839/killContainers
time="2023-01-30T10:27:50.203Z" level=info msg="Queueing Failed workflow argoci/demo-hsqz4 for delete in 168h0m0s due to TTL"
time="2023-01-30T10:27:55.144Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-cleanup-1813653132/deletePod
time="2023-01-30T10:28:10.031Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-cleanup-1813653132/killContainers
time="2023-01-30T10:28:41.285Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.286Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=0 workflow=demo-hsqz4
time="2023-01-30T10:28:41.286Z" level=info msg="All of node demo-hsqz4.c dependencies [a] completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.287Z" level=info msg="Pod node demo-hsqz4-1672856982 initialized Pending" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.306Z" level=info msg="Created pod: demo-hsqz4.c (demo-hsqz4-fail-test-1672856982)" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.306Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.306Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:41.324Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862607629 workflow=demo-hsqz4
time="2023-01-30T10:28:51.308Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:51.310Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=1 workflow=demo-hsqz4
time="2023-01-30T10:28:51.310Z" level=info msg="task-result changed" namespace=argoci nodeID=demo-hsqz4-1672856982 workflow=demo-hsqz4
time="2023-01-30T10:28:51.310Z" level=info msg="node changed" namespace=argoci new.message= new.phase=Running new.progress=0/1 nodeID=demo-hsqz4-1672856982 old.message= old.phase=Pending old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:28:51.310Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:51.310Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:28:51.315Z" level=info msg="cleaning up pod" action=terminateContainers key=argoci/demo-hsqz4-fail-test-1672856982/terminateContainers
time="2023-01-30T10:28:51.324Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=862607766 workflow=demo-hsqz4
time="2023-01-30T10:29:02.867Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.869Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=1 workflow=demo-hsqz4
time="2023-01-30T10:29:02.869Z" level=info msg="node changed" namespace=argoci new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=demo-hsqz4-1672856982 old.message= old.phase=Running old.progress=0/1 workflow=demo-hsqz4
time="2023-01-30T10:29:02.869Z" level=info msg="Outbound nodes of demo-hsqz4 set to [demo-hsqz4-1689634601 demo-hsqz4-1672856982 demo-hsqz4-1723189839 demo-hsqz4-1706412220]" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.869Z" level=info msg="node demo-hsqz4 phase Running -> Failed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.869Z" level=info msg="node demo-hsqz4 finished: 2023-01-30 10:29:02.869989182 +0000 UTC" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Checking daemoned children of demo-hsqz4" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Running OnExit handler: cleanup" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Updated phase Running -> Failed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Marking workflow completed" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Marking workflow as pending archiving" namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.870Z" level=info msg="Checking daemoned children of " namespace=argoci workflow=demo-hsqz4
time="2023-01-30T10:29:02.876Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-1340600742-agent/deletePod
time="2023-01-30T10:29:02.886Z" level=info msg="Workflow update successful" namespace=argoci phase=Failed resourceVersion=862607942 workflow=demo-hsqz4
time="2023-01-30T10:29:02.933Z" level=info msg="archiving workflow" namespace=argoci uid=3719303d-fee6-4419-8d06-a068474027e9 workflow=demo-hsqz4
time="2023-01-30T10:29:02.971Z" level=info msg="Queueing Failed workflow argoci/demo-hsqz4 for delete in 168h0m0s due to TTL"
time="2023-01-30T10:29:07.934Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-hsqz4-fail-test-1672856982/deletePod
time="2023-01-30T10:29:21.316Z" level=info msg="cleaning up pod" action=killContainers key=argoci/demo-hsqz4-fail-test-1672856982/killContainers

Logs from in your workflow's wait container

N/A

Jan 30 '23 10:01 RoryDoherty

argo-workflows argo-workflows copied to clipboard

Retry does not work correctly with a DAG when there are steps which have dependencies

Pre-requisites

What happened/what you expected to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

Logs from in your workflow's wait container

argo-workflows
argo-workflows copied to clipboard