kfp-tekton
kfp-tekton copied to clipboard
Task result not passed to nested range loop
/kind bug
What steps did you take and what happened: [A clear and concise description of what the bug is.]
The following DSL generates Tekton that does not pass task results to the inner range loop pipeline-84a66-for-loop-4 to
parameter
from kfp import dsl, components
from kfp.components import load_component_from_text
from kfp_tekton.tekton import Loop
op1_yaml = '''\
name: 'my-in-coop1'
inputs:
- {name: item, type: Integer}
- {name: param, type: Integer}
implementation:
container:
image: library/bash:4.4.23
command: ['sh', '-c']
args:
- |
set -e
echo op1 "$0" "$1"
- {inputValue: item}
- {inputValue: param}
'''
@dsl.pipeline(name='pipeline')
def pipeline():
cel_run_bash_script = load_component_from_text(r"""
name: cel-run-bash-script
inputs: []
outputs:
- {name: to_1, type: Integer}
implementation:
container:
image: aipipeline/cel-eval:latest
command: [cel]
args: [--apiVersion, custom.tekton.dev/v1alpha1, --kind, VariableStore, --name,
1234567-var-store, --lit_0, 'a', --taskSpec,
'{}']
fileOutputs: {}
""")()
with Loop.range(start=1, step=1, end=cel_run_bash_script.output):
with Loop.range(start=1, step=1, end=cel_run_bash_script.output):
op1_template = components.load_component_from_text(op1_yaml)
op1 = op1_template(1, 2)
if __name__ == '__main__':
from kfp_tekton.compiler import TektonCompiler as Compiler
from kfp_tekton.compiler.pipeline_utils import TektonPipelineConf
tekton_pipeline_conf = TektonPipelineConf()
tekton_pipeline_conf.set_tekton_inline_spec(True)
tekton_pipeline_conf.set_resource_in_separate_yaml(True)
Compiler().compile(pipeline, __file__.replace('.py', '.yaml'), tekton_pipeline_conf=tekton_pipeline_conf)
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipeline
annotations:
tekton.dev/output_artifacts: '{}'
tekton.dev/input_artifacts: '{}'
tekton.dev/artifact_bucket: mlpipeline
tekton.dev/artifact_endpoint: minio-service.kubeflow:9000
tekton.dev/artifact_endpoint_scheme: http://
tekton.dev/artifact_items: '{"my-in-coop1": []}'
sidecar.istio.io/inject: "false"
tekton.dev/template: ''
pipelines.kubeflow.org/big_data_passing_format: $(workspaces.$TASK_NAME.path)/artifacts/$ORIG_PR_NAME/$TASKRUN_NAME/$TASK_PARAM_NAME
pipelines.kubeflow.org/pipeline_spec: '{"name": "pipeline"}'
labels:
pipelines.kubeflow.org/pipelinename: ''
pipelines.kubeflow.org/generation: ''
spec:
pipelineSpec:
tasks:
- name: cel-run-bash-script
params:
- name: lit_0
value: a
taskSpec:
apiVersion: custom.tekton.dev/v1alpha1
kind: VariableStore
spec: {}
metadata:
labels:
pipelines.kubeflow.org/cache_enabled: "true"
annotations:
pipelines.kubeflow.org/component_spec_digest: '{"name": "cel-run-bash-script",
"outputs": [{"name": "to_1", "type": "Integer"}], "version": "cel-run-bash-script@sha256=973232bc30856bbd7dc5fc280eb165e2f4ceface92c22e762f9ebd7b860e5be2"}'
- name: pipeline-84a66-for-loop-2
params:
- name: cel-run-bash-script-to-1
value: $(tasks.cel-run-bash-script.results.to-1)
- name: from
value: '1'
- name: step
value: '1'
- name: to
value: $(tasks.cel-run-bash-script.results.to-1)
taskSpec:
apiVersion: custom.tekton.dev/v1alpha1
kind: PipelineLoop
spec:
pipelineSpec:
params:
- name: cel-run-bash-script-to-1
type: string
- name: loop-item-param-1
type: string
tasks:
- name: pipeline-84a66-for-loop-4
params:
- name: from
value: '1'
- name: step
value: '1'
taskSpec:
apiVersion: custom.tekton.dev/v1alpha1
kind: PipelineLoop
spec:
pipelineSpec:
params:
- name: loop-item-param-3
type: string
tasks:
- name: my-in-coop1
taskSpec:
steps:
- name: main
args:
- |
set -e
echo op1 "$0" "$1"
- '1'
- '2'
command:
- sh
- -c
image: library/bash:4.4.23
metadata:
labels:
pipelines.kubeflow.org/cache_enabled: "true"
annotations:
pipelines.kubeflow.org/component_spec_digest: '{"name":
"my-in-coop1", "outputs": [], "version": "my-in-coop1@sha256=8ccab3a28a39a406554d964865f2ccb0aed854a43b6de827f613eff2bccd6f8f"}'
iterateNumeric: loop-item-param-3
metadata:
labels:
pipelines.kubeflow.org/cache_enabled: "true"
iterateNumeric: loop-item-param-1
metadata:
labels:
pipelines.kubeflow.org/cache_enabled: "true"
This is a follow-up to the issue https://github.com/kubeflow/kfp-tekton/issues/1350. Passing task result output to nested loop was fixed for list-based loops but it still does not work for range loops
What did you expect to happen:
Additional information: [Miscellaneous information that will assist in solving the issue.]
Environment:
- Python Version (use
python --version
): - SDK Version:
- Tekton Version (use
tkn version
): - Kubernetes Version (use
kubectl version
): - OS (e.g. from
/etc/os-release
):