kfp-tekton icon indicating copy to clipboard operation
kfp-tekton copied to clipboard

Task result not passed to nested range loop

Open typeness opened this issue 7 months ago • 0 comments

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

The following DSL generates Tekton that does not pass task results to the inner range loop pipeline-84a66-for-loop-4 to parameter

from kfp import dsl, components
from kfp.components import load_component_from_text
from kfp_tekton.tekton import Loop

op1_yaml = '''\
name: 'my-in-coop1'
inputs:
- {name: item, type: Integer}
- {name: param, type: Integer}
implementation:
    container:
        image: library/bash:4.4.23
        command: ['sh', '-c']
        args:
        - |
          set -e
          echo op1 "$0" "$1"
        - {inputValue: item}
        - {inputValue: param}
'''




@dsl.pipeline(name='pipeline')
def pipeline():
    cel_run_bash_script = load_component_from_text(r"""
            name: cel-run-bash-script
            inputs: []
            outputs:
            - {name: to_1, type: Integer}
            implementation:
              container:
                image: aipipeline/cel-eval:latest
                command: [cel]
                args: [--apiVersion, custom.tekton.dev/v1alpha1, --kind, VariableStore, --name,
                  1234567-var-store, --lit_0, 'a', --taskSpec,
                  '{}']
                fileOutputs: {}
        """)()
    with Loop.range(start=1, step=1, end=cel_run_bash_script.output):
        with Loop.range(start=1, step=1, end=cel_run_bash_script.output):
            op1_template = components.load_component_from_text(op1_yaml)
            op1 = op1_template(1, 2)




if __name__ == '__main__':
  from kfp_tekton.compiler import TektonCompiler as Compiler
  from kfp_tekton.compiler.pipeline_utils import TektonPipelineConf
  tekton_pipeline_conf = TektonPipelineConf()
  tekton_pipeline_conf.set_tekton_inline_spec(True)
  tekton_pipeline_conf.set_resource_in_separate_yaml(True)
  Compiler().compile(pipeline, __file__.replace('.py', '.yaml'), tekton_pipeline_conf=tekton_pipeline_conf)


apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipeline
  annotations:
    tekton.dev/output_artifacts: '{}'
    tekton.dev/input_artifacts: '{}'
    tekton.dev/artifact_bucket: mlpipeline
    tekton.dev/artifact_endpoint: minio-service.kubeflow:9000
    tekton.dev/artifact_endpoint_scheme: http://
    tekton.dev/artifact_items: '{"my-in-coop1": []}'
    sidecar.istio.io/inject: "false"
    tekton.dev/template: ''
    pipelines.kubeflow.org/big_data_passing_format: $(workspaces.$TASK_NAME.path)/artifacts/$ORIG_PR_NAME/$TASKRUN_NAME/$TASK_PARAM_NAME
    pipelines.kubeflow.org/pipeline_spec: '{"name": "pipeline"}'
  labels:
    pipelines.kubeflow.org/pipelinename: ''
    pipelines.kubeflow.org/generation: ''
spec:
  pipelineSpec:
    tasks:
    - name: cel-run-bash-script
      params:
      - name: lit_0
        value: a
      taskSpec:
        apiVersion: custom.tekton.dev/v1alpha1
        kind: VariableStore
        spec: {}
        metadata:
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "cel-run-bash-script",
              "outputs": [{"name": "to_1", "type": "Integer"}], "version": "cel-run-bash-script@sha256=973232bc30856bbd7dc5fc280eb165e2f4ceface92c22e762f9ebd7b860e5be2"}'
    - name: pipeline-84a66-for-loop-2
      params:
      - name: cel-run-bash-script-to-1
        value: $(tasks.cel-run-bash-script.results.to-1)
      - name: from
        value: '1'
      - name: step
        value: '1'
      - name: to
        value: $(tasks.cel-run-bash-script.results.to-1)
      taskSpec:
        apiVersion: custom.tekton.dev/v1alpha1
        kind: PipelineLoop
        spec:
          pipelineSpec:
            params:
            - name: cel-run-bash-script-to-1
              type: string
            - name: loop-item-param-1
              type: string
            tasks:
            - name: pipeline-84a66-for-loop-4
              params:
              - name: from
                value: '1'
              - name: step
                value: '1'
              taskSpec:
                apiVersion: custom.tekton.dev/v1alpha1
                kind: PipelineLoop
                spec:
                  pipelineSpec:
                    params:
                    - name: loop-item-param-3
                      type: string
                    tasks:
                    - name: my-in-coop1
                      taskSpec:
                        steps:
                        - name: main
                          args:
                          - |
                            set -e
                            echo op1 "$0" "$1"
                          - '1'
                          - '2'
                          command:
                          - sh
                          - -c
                          image: library/bash:4.4.23
                        metadata:
                          labels:
                            pipelines.kubeflow.org/cache_enabled: "true"
                          annotations:
                            pipelines.kubeflow.org/component_spec_digest: '{"name":
                              "my-in-coop1", "outputs": [], "version": "my-in-coop1@sha256=8ccab3a28a39a406554d964865f2ccb0aed854a43b6de827f613eff2bccd6f8f"}'
                  iterateNumeric: loop-item-param-3
                metadata:
                  labels:
                    pipelines.kubeflow.org/cache_enabled: "true"
          iterateNumeric: loop-item-param-1
        metadata:
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"

This is a follow-up to the issue https://github.com/kubeflow/kfp-tekton/issues/1350. Passing task result output to nested loop was fixed for list-based loops but it still does not work for range loops

What did you expect to happen:

Additional information: [Miscellaneous information that will assist in solving the issue.]

Environment:

  • Python Version (use python --version):
  • SDK Version:
  • Tekton Version (use tkn version):
  • Kubernetes Version (use kubectl version):
  • OS (e.g. from /etc/os-release):

typeness avatar Jun 27 '24 10:06 typeness