Nested loops pipeline does not run from notebook

[blublinsky]

/kind bug

Converted nested loops python code to notebook, replacung

if name == 'main': from kfp_tekton.compiler import TektonCompiler TektonCompiler().compile(pipeline, file.replace('.py', '.yaml'))

with:

Specify Kubeflow Pipeline Host

host=None

Submit a pipeline run using the KFP Tekton client.

TektonClient(host=host).create_run_from_pipeline_func(pipeline, arguments = {})

While Python version works fine, notebook version fails

Environment:

• Python Version 3.7
• SDK Version latest
• Tekton Version (use tkn version): v0.24
• Kubernetes Version (use kubectl version): OpenShift 4.6.28_1543 on IBM cloud
• OS (e.g. from /etc/os-release): OpenShift 4.6.28_1543 on IBM cloud

[Tomcli]

Currently you have to create the Tekton loop resources before calling the create_run_from_pipeline_func function. To generate the loop resources, you can run the first part of the nested loop instruction https://github.com/kubeflow/kfp-tekton/tree/master/samples/nested-loops#instructions

We introduced a new feature in Tekton v0.25 to embed all the loop resources into a single pipelinerun. PR #633 will support this and will be coming very soon.

[blublinsky]

The strange thing, is that when running Python code, It generates a single file containing everything, so I was assuming it will work in notebook as well. So how is invocation should look like in the notebook? I am not sure how to implement it. Can you sketch the code, please?

[blublinsky]

@Tomcli, I think I found the issue. for pipelines v0.8.0, It works with kfp-tekton==0.6.0 but not python library 0.8.1

[Tomcli]

On 0.6.0, the nested loop feature wasn't implemented, so you are probably running an incomplete pipeline. This issue should be fixed in the upcoming kfp-tekton 0.9.0 as we are working on some enhancement to inline all the Tekton resources into one yaml

Here are the WIP PRs #633 #636 #635

[Tomcli]

As for now, you have to follow this instruction to generate the Tekton resources separately because we just added the inline enhancement to the Tekton community few weeks ago in Tekton v0.25.0.

https://github.com/kubeflow/kfp-tekton/tree/master/samples/nested-loops#instructions

[Tomcli]

FYI this issue should be fixed if you run with our 0.9.0 kfp-tekton release with Tekton 0.25+ https://github.com/kubeflow/kfp-tekton/blob/master/guides/kfp_tekton_install.md#standalone-kubeflow-pipelines-with-tekton-backend-deployment

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Tomcli]

/close

This is fixed since KFP-Tekton 0.9 release

[google-oss-prow[bot]]

@Tomcli: Closing this issue.

In response to this:

/close

This is fixed since KFP-Tekton 0.9 release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.