argo-workflows
argo-workflows copied to clipboard
Published
20 hours ago •
argoproj
argoproj
Unable to resolve artifact by using its own artifact storage sometimes
Checklist
- [x] Double-checked my configuration.
- [x] Tested using the latest version.
- [ ] Used the Emissary executor.
Summary
What happened/what you expected to happen?
I expected the dag workflow is ok all the time instead of sometimes. I resubmit the workflow, but it just works sometimes, not all the time. It shows "Unable to resolve: "tasks.generate-artifact.outputs.artifacts.hello-art"".
What version are you running?
v3.3.8
Diagnostics
Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifact-passing-dag-
spec:
entrypoint: dag
templates:
- name: dag
dag:
tasks:
# generate output artifacts of parallel pods
- name: generate-artifact
template: whalesay
# aggregate
- name: consume-artifact
depends: "generate-artifact"
template: print-message
arguments:
artifacts:
- name: message
from: "{{tasks.generate-artifact.outputs.artifacts.hello-art}}"
- name: whalesay
container:
image: docker/whalesay:latest
command: [sh, -c]
args: ["cowsay hello world | tee /tmp/hello_world.txt"]
volumeMounts:
- name: out
mountPath: /tmp
volumes:
- name: out
emptyDir: { }
outputs:
artifacts:
# generate hello-art artifact from /tmp/hello_world.txt
# artifacts can be directories as well as files
- name: hello-art
path: /tmp/hello_world.txt
- name: print-message
inputs:
artifacts:
# unpack the message input artifact
# and put it at /tmp/message
- name: message
path: /tmp/message
container:
image: alpine:latest
command: [sh, -c]
args: ["cat /tmp/message"]
# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2022-06-29T07:55:25.905Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=56223314 workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.891Z" level=info msg="Processing workflow" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.891Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.891Z" level=info msg="task-result changed" namespace=argo nodeID=artifact-passing-dag-tn7lz-3189522870 workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.892Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=artifact-passing-dag-tn7lz-3189522870 old.message= old.phase=Pending old.progress=0/1
time="2022-06-29T07:55:35.892Z" level=info msg="All of node artifact-passing-dag-tn7lz.consume-artifact dependencies [generate-artifact] completed" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.892Z" level=info msg="Pod node artifact-passing-dag-tn7lz-2741019177 initialized Pending" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.897Z" level=info msg="Created pod: artifact-passing-dag-tn7lz.consume-artifact (artifact-passing-dag-tn7lz-2741019177)" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.897Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.897Z" level=info msg=reconcileAgentPod namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.902Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"artifact-passing-dag-tn7lz\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.902Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=artifact-passing-dag-tn7lz
time="2022-06-29T07:55:35.905Z" level=info msg="Failed to re-apply update" error="must never update completed workflows" namespace=argo workflow=artifact-passing-dag-tn7lz
# If the workflow's pods have not been created, you can skip the rest of the diagnostics.
# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: 047f4a4bb206bbfb3a4a0da0441eeec1c510959661b48ca1e4a63dbd1f437f86
cni.projectcalico.org/podIP: ""
cni.projectcalico.org/podIPs: ""
kubectl.kubernetes.io/default-container: main
pipelines.kubeflow.org/metadata_input_artifact_ids: '[]'
workflows.argoproj.io/node-id: artifact-passing-dag-tn7lz-2741019177
workflows.argoproj.io/node-name: artifact-passing-dag-tn7lz.consume-artifact
creationTimestamp: "2022-06-29T07:55:35Z"
labels:
pipelines.kubeflow.org/metadata_context_id: "144"
pipelines.kubeflow.org/metadata_execution_id: "222"
workflows.argoproj.io/completed: "false"
workflows.argoproj.io/workflow: artifact-passing-dag-tn7lz
name: artifact-passing-dag-tn7lz-2741019177
namespace: argo
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Workflow
name: artifact-passing-dag-tn7lz
uid: 799360ab-bfdf-4ba4-8a93-74159ee01894
resourceVersion: "56223828"
uid: 318cf9fe-9488-45d0-8fc0-eecc378fe73c
spec:
containers:
- command:
- argoexec
- wait
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: artifact-passing-dag-tn7lz
- name: ARGO_CONTAINER_NAME
value: wait
- name: ARGO_TEMPLATE
value: '{"name":"print-message","inputs":{"artifacts":[{"name":"message","path":"/tmp/message","s3":{"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/hello-art.tgz"}}]},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["sh","-c"],"args":["cat
/tmp/message"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-2741019177"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-2741019177
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:latest
imagePullPolicy: Always
name: wait
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/my-minio-cred
name: my-minio-cred
readOnly: true
- mountPath: /mainctrfs/tmp/message
name: input-artifacts
subPath: message
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-srlrq
readOnly: true
- args:
- cat /tmp/message
command:
- /var/run/argo/argoexec
- emissary
- --
- sh
- -c
env:
- name: ARGO_CONTAINER_NAME
value: main
- name: ARGO_TEMPLATE
value: '{"name":"print-message","inputs":{"artifacts":[{"name":"message","path":"/tmp/message","s3":{"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/hello-art.tgz"}}]},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["sh","-c"],"args":["cat
/tmp/message"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-2741019177"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-2741019177
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: alpine:latest
imagePullPolicy: Always
name: main
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/message
name: input-artifacts
subPath: message
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-srlrq
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- command:
- argoexec
- init
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: artifact-passing-dag-tn7lz
- name: ARGO_CONTAINER_NAME
value: init
- name: ARGO_TEMPLATE
value: '{"name":"print-message","inputs":{"artifacts":[{"name":"message","path":"/tmp/message","s3":{"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/hello-art.tgz"}}]},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["sh","-c"],"args":["cat
/tmp/message"],"resources":{}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-2741019177"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-2741019177
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:latest
imagePullPolicy: Always
name: init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/my-minio-cred
name: my-minio-cred
readOnly: true
- mountPath: /argo/inputs/artifacts
name: input-artifacts
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-srlrq
readOnly: true
nodeName: ip-10-100-5-82
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: var-run-argo
- name: my-minio-cred
secret:
defaultMode: 420
items:
- key: accesskey
path: accesskey
- key: secretkey
path: secretkey
secretName: my-minio-cred
- emptyDir: {}
name: input-artifacts
- name: kube-api-access-srlrq
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:39Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:43Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:43Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:35Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://a153618df65c1e16b5364dbd527eb9f333ee01bb11af558031159b24cfcbce84
image: alpine:latest
imageID: docker-pullable://alpine@sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
lastState: {}
name: main
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://a153618df65c1e16b5364dbd527eb9f333ee01bb11af558031159b24cfcbce84
exitCode: 0
finishedAt: "2022-06-29T07:55:42Z"
reason: Completed
startedAt: "2022-06-29T07:55:41Z"
- containerID: docker://1965a50d77343607798d0e7811caeceee506fa6b5c4b9276ba5a71cc0b0369ca
image: quay.io/argoproj/argoexec:latest
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:b4257735f9ee1d9be07e7d5c24919c18dc6969ff76b570305e5c098a9faac135
lastState: {}
name: wait
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://1965a50d77343607798d0e7811caeceee506fa6b5c4b9276ba5a71cc0b0369ca
exitCode: 0
finishedAt: "2022-06-29T07:55:43Z"
reason: Completed
startedAt: "2022-06-29T07:55:40Z"
hostIP: 10.100.5.82
initContainerStatuses:
- containerID: docker://dc265ed705653bd3ced3fa32b998bed80ea993d4aa6fa08061182de3ec2c0c56
image: quay.io/argoproj/argoexec:latest
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:b4257735f9ee1d9be07e7d5c24919c18dc6969ff76b570305e5c098a9faac135
lastState: {}
name: init
ready: true
restartCount: 0
state:
terminated:
containerID: docker://dc265ed705653bd3ced3fa32b998bed80ea993d4aa6fa08061182de3ec2c0c56
exitCode: 0
finishedAt: "2022-06-29T07:55:38Z"
reason: Completed
startedAt: "2022-06-29T07:55:38Z"
phase: Succeeded
podIP: 10.244.207.237
podIPs:
- ip: 10.244.207.237
qosClass: BestEffort
startTime: "2022-06-29T07:55:35Z"
- apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: 26c116adb1fb84bdfe9397ddddcaa160ca51acd2ad26006769b3c218942a667b
cni.projectcalico.org/podIP: ""
cni.projectcalico.org/podIPs: ""
kubectl.kubernetes.io/default-container: main
pipelines.kubeflow.org/metadata_input_artifact_ids: '[]'
pipelines.kubeflow.org/metadata_output_artifact_ids: '[]'
workflows.argoproj.io/node-id: artifact-passing-dag-tn7lz-3189522870
workflows.argoproj.io/node-name: artifact-passing-dag-tn7lz.generate-artifact
creationTimestamp: "2022-06-29T07:55:25Z"
labels:
pipelines.kubeflow.org/metadata_context_id: "144"
pipelines.kubeflow.org/metadata_execution_id: "221"
pipelines.kubeflow.org/metadata_written: "true"
workflows.argoproj.io/completed: "true"
workflows.argoproj.io/workflow: artifact-passing-dag-tn7lz
name: artifact-passing-dag-tn7lz-3189522870
namespace: argo
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Workflow
name: artifact-passing-dag-tn7lz
uid: 799360ab-bfdf-4ba4-8a93-74159ee01894
resourceVersion: "56223611"
uid: dd2f3872-0db1-489a-b010-389dbae57ddd
spec:
containers:
- command:
- argoexec
- wait
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: artifact-passing-dag-tn7lz
- name: ARGO_CONTAINER_NAME
value: wait
- name: ARGO_TEMPLATE
value: '{"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"hello-art","path":"/tmp/hello_world.txt"}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay
hello world | tee /tmp/hello_world.txt"],"resources":{},"volumeMounts":[{"name":"out","mountPath":"/tmp"}]},"volumes":[{"name":"out","emptyDir":{}}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-3189522870
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:latest
imagePullPolicy: Always
name: wait
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/my-minio-cred
name: my-minio-cred
readOnly: true
- mountPath: /mainctrfs/tmp
name: out
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zvfjn
readOnly: true
- args:
- cowsay hello world | tee /tmp/hello_world.txt
command:
- /var/run/argo/argoexec
- emissary
- --
- sh
- -c
env:
- name: ARGO_CONTAINER_NAME
value: main
- name: ARGO_TEMPLATE
value: '{"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"hello-art","path":"/tmp/hello_world.txt"}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay
hello world | tee /tmp/hello_world.txt"],"resources":{},"volumeMounts":[{"name":"out","mountPath":"/tmp"}]},"volumes":[{"name":"out","emptyDir":{}}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-3189522870
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: docker/whalesay:latest
imagePullPolicy: Always
name: main
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: out
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zvfjn
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- command:
- argoexec
- init
- --loglevel
- info
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_CONTAINER_RUNTIME_EXECUTOR
value: emissary
- name: GODEBUG
value: x509ignoreCN=0
- name: ARGO_WORKFLOW_NAME
value: artifact-passing-dag-tn7lz
- name: ARGO_CONTAINER_NAME
value: init
- name: ARGO_TEMPLATE
value: '{"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"hello-art","path":"/tmp/hello_world.txt"}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay
hello world | tee /tmp/hello_world.txt"],"resources":{},"volumeMounts":[{"name":"out","mountPath":"/tmp"}]},"volumes":[{"name":"out","emptyDir":{}}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"myminio.minio:9000","bucket":"my-argo-bucket","insecure":true,"accessKeySecret":{"name":"my-minio-cred","key":"accesskey"},"secretKeySecret":{"name":"my-minio-cred","key":"secretkey"},"key":"artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870"}}}'
- name: ARGO_NODE_ID
value: artifact-passing-dag-tn7lz-3189522870
- name: ARGO_INCLUDE_SCRIPT_OUTPUT
value: "false"
- name: ARGO_DEADLINE
value: "0001-01-01T00:00:00Z"
- name: ARGO_PROGRESS_FILE
value: /var/run/argo/progress
- name: ARGO_PROGRESS_PATCH_TICK_DURATION
value: 1m0s
- name: ARGO_PROGRESS_FILE_TICK_DURATION
value: 3s
image: quay.io/argoproj/argoexec:latest
imagePullPolicy: Always
name: init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /argo/secret/my-minio-cred
name: my-minio-cred
readOnly: true
- mountPath: /var/run/argo
name: var-run-argo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zvfjn
readOnly: true
nodeName: ip-10-100-5-82
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: var-run-argo
- emptyDir: {}
name: out
- name: my-minio-cred
secret:
defaultMode: 420
items:
- key: accesskey
path: accesskey
- key: secretkey
path: secretkey
secretName: my-minio-cred
- name: kube-api-access-zvfjn
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:29Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:33Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:33Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-06-29T07:55:25Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://b850101ee42791a6015a10ee91f4454127777112153f68ab3dd7ee216ef603d0
image: docker/whalesay:latest
imageID: docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
lastState: {}
name: main
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://b850101ee42791a6015a10ee91f4454127777112153f68ab3dd7ee216ef603d0
exitCode: 0
finishedAt: "2022-06-29T07:55:33Z"
reason: Completed
startedAt: "2022-06-29T07:55:32Z"
- containerID: docker://42515e5612664b8656a74e92abc73097edace94d8b577f1b5b639fa2e9816741
image: quay.io/argoproj/argoexec:latest
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:b4257735f9ee1d9be07e7d5c24919c18dc6969ff76b570305e5c098a9faac135
lastState: {}
name: wait
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: docker://42515e5612664b8656a74e92abc73097edace94d8b577f1b5b639fa2e9816741
exitCode: 0
finishedAt: "2022-06-29T07:55:34Z"
reason: Completed
startedAt: "2022-06-29T07:55:30Z"
hostIP: 10.100.5.82
initContainerStatuses:
- containerID: docker://529a336df691078946ce1cb7abecf3d556f3a46e37bd3268bc0c8026db07d309
image: quay.io/argoproj/argoexec:latest
imageID: docker-pullable://quay.io/argoproj/argoexec@sha256:b4257735f9ee1d9be07e7d5c24919c18dc6969ff76b570305e5c098a9faac135
lastState: {}
name: init
ready: true
restartCount: 0
state:
terminated:
containerID: docker://529a336df691078946ce1cb7abecf3d556f3a46e37bd3268bc0c8026db07d309
exitCode: 0
finishedAt: "2022-06-29T07:55:28Z"
reason: Completed
startedAt: "2022-06-29T07:55:28Z"
phase: Succeeded
podIP: 10.244.207.230
podIPs:
- ip: 10.244.207.230
qosClass: BestEffort
startTime: "2022-06-29T07:55:25Z"
kind: List
metadata:
resourceVersion: ""
selfLink: ""
# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
time="2022-06-29T07:55:43.345Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-06-29T07:55:43.345Z" level=info msg="Deadline monitor stopped"
time="2022-06-29T07:55:43.345Z" level=info msg="Alloc=6413 TotalAlloc=12603 Sys=23762 NumGC=4 Goroutines=11"
time="2022-06-29T07:55:33.929Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/main.log"
time="2022-06-29T07:55:33.929Z" level=info msg="Creating minio client using static credentials" endpoint="myminio.minio:9000"
time="2022-06-29T07:55:33.929Z" level=info msg="Saving file to s3" bucket=my-argo-bucket endpoint="myminio.minio:9000" key=artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-06-29T07:55:33.979Z" level=info msg="Save artifact" artifactName=main-logs duration=50.220981ms error="<nil>" key=artifact-passing-dag-tn7lz/artifact-passing-dag-tn7lz-3189522870/main.log
time="2022-06-29T07:55:33.980Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-06-29T07:55:33.980Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-06-29T07:55:33.987Z" level=info msg="Create workflowtaskresults 201"
time="2022-06-29T07:55:33.988Z" level=info msg="Deadline monitor stopped"
time="2022-06-29T07:55:33.988Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-06-29T07:55:33.988Z" level=info msg="Alloc=8535 TotalAlloc=13733 Sys=23506 NumGC=4 Goroutines=11"
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@chenbodeng719 Hello, I have run this workflow a few times now on master branch and I've been unable to see this error reproduce. Could you try running again and see how often this error appears?
@dpadhiar You can click the resubmit the workflow multi times in a short time. It will happen sometimes. I used the namespace-install.yaml in argo namespace.
@dpadhiar You can click the resubmit the workflow multi times in a short time. It will happen sometimes. I used the namespace-install.yaml in argo namespace.
After more attempts today I have been unable to replicate the error. However, we assume this is a small fix.
@sarabala1979 Could you point to where the fix for this would be in the code?
@dpadhiar Anyway, it looks like if you use its own artifact storage to pass data, it's not stable. Sometimes it can't resolve the artifact source. So I change my way to use volumeClaimTemplates for artifact tmp location. There is a demo of it. example Hope it can help others.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been closed due to inactivity. Feel free to re-open if you still encounter this issue.