pipelines
pipelines copied to clipboard
Published
20 hours ago •
kubeflow
kubeflow
[bug] executor attempts to kill already terminated kfp-launcher init container
Environment
- How do you deploy Kubeflow Pipelines (KFP)? standalone-deployment
- KFP version: 1.8.2
- KFP SDK version: 1.8.2
Steps to reproduce
The KFPv2 pipelines are completed, but errors occur as shown below, it seems to be a problem caused by attempting to terminate the already terminated the kfp-launcher init container.
$ kubectl -n kubeflow-pipeline logs -f pipeline-test-1-gtvhr-2680780413 -c wait
time="2022-07-12T06:48:44.957Z" level=info msg="Starting Workflow Executor" executorType=docker version=v3.2.3
time="2022-07-12T06:48:44.961Z" level=info msg="Creating a docker executor"
time="2022-07-12T06:48:44.961Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=kubeflow-pipeline podName=pipeline-test-1-gtvhr-2680780413 template="{\"name\":\"get-data\",\"inputs\":{\"parameters\":[{\"name\":\"pipeline-name\",\"value\":\"pipeline/pipeline-test-1\"},{\"name\":\"pipeline-root\",\"value\":\"\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"get-data-dataset_test\",\"path\":\"/tmp/outputs/dataset_test/data\"},{\"name\":\"get-data-dataset_train\",\"path\":\"/tmp/outputs/dataset_train/data\"}]},\"metadata\":{\"annotations\":{\"pipelines.kubeflow.org/component_ref\":\"{}\",\"pipelines.kubeflow.org/v2_component\":\"true\",\"sidecar.istio.io/inject\":\"false\"},\"labels\":{\"pipelines.kubeflow.org/cache_enabled\":\"true\",\"pipelines.kubeflow.org/enable_caching\":\"false\",\"pipelines.kubeflow.org/kfp_sdk_version\":\"1.8.2\",\"pipelines.kubeflow.org/pipeline-sdk-type\":\"kfp\",\"pipelines.kubeflow.org/v2_component\":\"true\"}},\"container\":{\"name\":\"\",\"image\":\"python:3.7\",\"command\":[\"/kfp-launcher/launch\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--runtime_info_json\",\"$(KFP_V2_RUNTIME_INFO)\",\"--container_image\",\"$(KFP_V2_IMAGE)\",\"--task_name\",\"get-data\",\"--pipeline_name\",\"pipeline/pipeline-test-1\",\"--run_id\",\"$(KFP_RUN_ID)\",\"--run_resource\",\"workflows.argoproj.io/$(WORKFLOW_ID)\",\"--namespace\",\"$(KFP_NAMESPACE)\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--pipeline_root\",\"\",\"--enable_caching\",\"$(ENABLE_CACHING)\",\"--\",\"--\"],\"args\":[\"sh\",\"-c\",\"(python3 -m ensurepip || python3 -m ensurepip --user) \\u0026\\u0026 (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'pandas' 'sklearn' 'kfp==1.8.2' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'pandas' 'sklearn' 'kfp==1.8.2' --user) \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main --component_module_path \\\"$program_path/ephemeral_component.py\\\" \\\"$@\\\"\\n\",\"\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef get_data(\\n dataset_train: Output[Dataset],\\n dataset_test: Output[Dataset]\\n\\n):\\n\\n from sklearn import datasets\\n from sklearn.model_selection import train_test_split as tts\\n import pandas as pd\\n # import some data to play with\\n\\n data_raw = datasets.load_breast_cancer()\\n data = pd.DataFrame(data_raw.data, columns=data_raw.feature_names)\\n data[\\\"target\\\"] = data_raw.target\\n\\n train, test = tts(data, test_size=0.3)\\n\\n train.to_csv(dataset_train.path)\\n test.to_csv(dataset_test.path)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"get_data\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"KFP_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"WORKFLOW_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.labels['workflows.argoproj.io/workflow']\"}}},{\"name\":\"KFP_RUN_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.labels['pipeline/runid']\"}}},{\"name\":\"ENABLE_CACHING\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.labels['pipelines.kubeflow.org/enable_caching']\"}}},{\"name\":\"KFP_V2_IMAGE\",\"value\":\"python:3.7\"},{\"name\":\"KFP_V2_RUNTIME_INFO\",\"value\":\"{\\\"inputParameters\\\": {}, \\\"inputArtifacts\\\": {}, \\\"outputParameters\\\": {}, \\\"outputArtifacts\\\": {\\\"dataset_test\\\": {\\\"schemaTitle\\\": \\\"system.Dataset\\\", \\\"instanceSchema\\\": \\\"\\\", \\\"schemaVersion\\\": \\\"0.0.1\\\", \\\"metadataPath\\\": \\\"/tmp/outputs/dataset_test/data\\\"}, \\\"dataset_train\\\": {\\\"schemaTitle\\\": \\\"system.Dataset\\\", \\\"instanceSchema\\\": \\\"\\\", \\\"schemaVersion\\\": \\\"0.0.1\\\", \\\"metadataPath\\\": \\\"/tmp/outputs/dataset_train/data\\\"}}}\"}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\"}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher:1.8.2\",\"command\":[\"launcher\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{},\"mirrorVolumeMounts\":true}],\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow-pipeline:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"clops-minio-artifact\",\"key\":\"accessKey\"},\"secretKeySecret\":{\"name\":\"clops-minio-artifact\",\"key\":\"secretKey\"},\"key\":\"pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413\"}}}" version="&Version{Version:v3.2.3,BuildDate:2021-10-27T01:09:16Z,GitCommit:e5dc961b7846efe0fe36ab3a0964180eaedd2672,GitTag:v3.2.3,GitTreeState:clean,GoVersion:go1.16.9,Compiler:gc,Platform:linux/amd64,}"
time="2022-07-12T06:48:44.961Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:44.961Z" level=info msg="Starting deadline monitor"
time="2022-07-12T06:48:45.006Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Created {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:45.006Z" level=info msg="mapped container name \"wait\" to container ID \"b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c\" (created at 2022-07-12 06:48:44 +0000 UTC, status Up)"
time="2022-07-12T06:48:45.006Z" level=info msg="mapped container name \"kfp-launcher\" to container ID \"3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b\" (created at 2022-07-12 06:48:43 +0000 UTC, status Exited)"
time="2022-07-12T06:48:45.006Z" level=info msg="mapped container name \"main\" to container ID \"e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c\" (created at 2022-07-12 06:48:44 +0000 UTC, status Created)"
time="2022-07-12T06:48:45.962Z" level=info msg="docker wait e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c"
time="2022-07-12T06:48:46.007Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:46.036Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:47.036Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:47.068Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:48.069Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:48.099Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:49.100Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:49.130Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:50.131Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:50.172Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:51.173Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:51.204Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:52.204Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:52.231Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:53.231Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:53.265Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:54.266Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:54.294Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:55.295Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:55.328Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:56.328Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:56.357Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:57.357Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:57.386Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:58.387Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:58.421Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:48:59.422Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:48:59.449Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:00.449Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:00.477Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:01.477Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:01.530Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:02.530Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:02.565Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:03.565Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:03.607Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:04.607Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:04.638Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:05.640Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:05.693Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:06.693Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:06.749Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:07.749Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:07.783Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:08.784Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:08.848Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:09.849Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:09.883Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:10.884Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:10.934Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:11.935Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:11.963Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:12.964Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:12.997Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:13.997Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:14.027Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:15.028Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:15.058Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Up {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:22.093Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:22.195Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Exited {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:22.195Z" level=info msg="Main container completed"
time="2022-07-12T06:49:22.195Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-07-12T06:49:22.195Z" level=info msg="Saving logs"
time="2022-07-12T06:49:22.195Z" level=info msg="[docker logs e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c]"
time="2022-07-12T06:49:22.247Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/main.log"
time="2022-07-12T06:49:22.247Z" level=info msg="Creating minio client using static credentials" endpoint="minio-service.kubeflow-pipeline:9000"
time="2022-07-12T06:49:22.247Z" level=info msg="Saving file to s3" bucket=mlpipeline endpoint="minio-service.kubeflow-pipeline:9000" key=pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-07-12T06:49:24.735Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-07-12T06:49:24.735Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-07-12T06:49:24.735Z" level=info msg="No output parameters"
time="2022-07-12T06:49:24.735Z" level=info msg="Saving output artifacts"
time="2022-07-12T06:49:24.735Z" level=info msg="Staging artifact: get-data-dataset_test"
time="2022-07-12T06:49:24.735Z" level=info msg="Copying /tmp/outputs/dataset_test/data from container base image layer to /tmp/argo/outputs/artifacts/get-data-dataset_test.tgz"
time="2022-07-12T06:49:24.735Z" level=info msg="Archiving main:/tmp/outputs/dataset_test/data to /tmp/argo/outputs/artifacts/get-data-dataset_test.tgz"
time="2022-07-12T06:49:24.735Z" level=info msg="sh -c docker cp -a e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c:/tmp/outputs/dataset_test/data - | gzip > /tmp/argo/outputs/artifacts/get-data-dataset_test.tgz"
time="2022-07-12T06:49:24.915Z" level=info msg="Archiving completed"
time="2022-07-12T06:49:24.915Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/get-data-dataset_test.tgz, key: pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/get-data-dataset_test.tgz"
time="2022-07-12T06:49:24.915Z" level=info msg="Creating minio client using static credentials" endpoint="minio-service.kubeflow-pipeline:9000"
time="2022-07-12T06:49:24.915Z" level=info msg="Saving file to s3" bucket=mlpipeline endpoint="minio-service.kubeflow-pipeline:9000" key=pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/get-data-dataset_test.tgz path=/tmp/argo/outputs/artifacts/get-data-dataset_test.tgz
time="2022-07-12T06:49:25.058Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:25.089Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Exited {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:26.424Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/get-data-dataset_test.tgz
time="2022-07-12T06:49:26.424Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/get-data-dataset_test.tgz"
time="2022-07-12T06:49:26.424Z" level=info msg="Staging artifact: get-data-dataset_train"
time="2022-07-12T06:49:26.424Z" level=info msg="Copying /tmp/outputs/dataset_train/data from container base image layer to /tmp/argo/outputs/artifacts/get-data-dataset_train.tgz"
time="2022-07-12T06:49:26.424Z" level=info msg="Archiving main:/tmp/outputs/dataset_train/data to /tmp/argo/outputs/artifacts/get-data-dataset_train.tgz"
time="2022-07-12T06:49:26.424Z" level=info msg="sh -c docker cp -a e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c:/tmp/outputs/dataset_train/data - | gzip > /tmp/argo/outputs/artifacts/get-data-dataset_train.tgz"
time="2022-07-12T06:49:26.559Z" level=info msg="Archiving completed"
time="2022-07-12T06:49:26.571Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/get-data-dataset_train.tgz, key: pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/get-data-dataset_train.tgz"
time="2022-07-12T06:49:26.571Z" level=info msg="Creating minio client using static credentials" endpoint="minio-service.kubeflow-pipeline:9000"
time="2022-07-12T06:49:26.571Z" level=info msg="Saving file to s3" bucket=mlpipeline endpoint="minio-service.kubeflow-pipeline:9000" key=pipeline-test-1-gtvhr/pipeline-test-1-gtvhr-2680780413/get-data-dataset_train.tgz path=/tmp/argo/outputs/artifacts/get-data-dataset_train.tgz
time="2022-07-12T06:49:28.158Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/get-data-dataset_train.tgz
time="2022-07-12T06:49:28.158Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/get-data-dataset_train.tgz"
time="2022-07-12T06:49:28.158Z" level=info msg="Annotating pod with output"
time="2022-07-12T06:49:28.211Z" level=info msg="Patch pods 200"
time="2022-07-12T06:49:28.218Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=kubeflow-pipeline --filter=label=io.kubernetes.pod.name=pipeline-test-1-gtvhr-2680780413"
time="2022-07-12T06:49:28.250Z" level=info msg="listed containers" containers="map[kfp-launcher:{3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b Exited {0 63793205323 <nil>}} main:{e9f48e33f389b70943f7a04eaed064a646265b4c832216a47bf6c7e20a6aca9c Exited {0 63793205324 <nil>}} wait:{b17b4bd21c798e1cb684b62b80435c92705737169f69e51a09e56e9c980a373c Up {0 63793205324 <nil>}}]"
time="2022-07-12T06:49:28.250Z" level=info msg="Killing sidecars [\"kfp-launcher\"]"
time="2022-07-12T06:49:28.252Z" level=info msg="Get pods 403"
time="2022-07-12T06:49:28.252Z" level=warning msg="Non-transient error: pods \"pipeline-test-1-gtvhr-2680780413\" is forbidden: User \"system:serviceaccount:kubeflow-pipeline:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"kubeflow-pipeline\""
time="2022-07-12T06:49:28.252Z" level=info msg="docker kill --signal TERM 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b"
time="2022-07-12T06:49:28.306Z" level=error msg="`docker kill --signal TERM 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b` failed: Error response from daemon: Cannot kill container: 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b: Container 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b is not running\n"
time="2022-07-12T06:49:28.306Z" level=warning msg="Ignored error from 'docker kill --signal TERM': Error response from daemon: Cannot kill container: 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b: Container 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b is not running"
time="2022-07-12T06:49:28.306Z" level=info msg="[docker wait 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b]"
time="2022-07-12T06:49:28.307Z" level=info msg="Timed out (0s) for containers to terminate gracefully. Killing forcefully"
time="2022-07-12T06:49:28.307Z" level=info msg="[docker kill --signal KILL 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b]"
time="2022-07-12T06:49:28.405Z" level=warning msg="Ignored error from 'docker kill --signal KILL': exit status 1"
time="2022-07-12T06:49:28.405Z" level=error msg="executor error: Error response from daemon: Cannot kill container: 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b: Container 3910ce0b5e1470c0f11ccc5d1e9895feec6e9bec565b7eb0cade8e3f053b843b is not running\ngithub.com/argoproj/argo-workflows/v3/errors.Wrap\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:88\ngithub.com/argoproj/argo-workflows/v3/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:71\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/docker.(*DockerExecutor).Kill\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/docker/docker.go:410\ngithub.com/argoproj/argo-workflows/v3/workflow/executor.(*WorkflowExecutor).KillSidecars\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/executor.go:989\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:35\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:72\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:18\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:860\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:902\nmain.main\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"
time="2022-07-12T06:49:28.405Z" level=info msg="Alloc=6364 TotalAlloc=22190 Sys=73297 NumGC=6 Goroutines=15"
Expected result
Materials and reference
- my workflow-controller-configmap
apiVersion: v1
data:
artifactRepository: |
archiveLogs: true
s3:
accessKeySecret:
key: accesskey
name: mlpipeline-minio-artifact
bucket: mlpipeline
endpoint: minio-service.kubeflow-pipeline:9000
insecure: true
secretKeySecret:
key: secretkey
name: mlpipeline-minio-artifact
containerRuntimeExecutor: docker
executor: |
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: "0.5"
memory: 512Mi
requests:
cpu: "0.01"
memory: 32Mi
parallelism: "50"
kind: ConfigMap
metadata:
name: workflow-controller-configmap
namespace: kubeflow-pipeline
- KFPv2 example I tried: https://github.com/kubeflow/pipelines/blob/master/samples/v2/lightweight_python_functions_v2_pipeline/lightweight_python_functions_v2_pipeline.py
Labels
/area backend /area sdk
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Hi @jazzsir , can you provide what kubernetes version you used? We are moving toward the emissary executor. Here is the doc for executor: https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@rimolive: Closing this issue.
In response to this:
Closing this issue. No activity for more than a year.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
![google-oss-prow[bot] avatar](https://avatars.githubusercontent.com/in/143327?v=4 "Published by a pub.dev verified publisher"){kind=small}