[sdk] Compilation with V2_COMPATIBLE mode fails when inputPath/outputPath name contains an empty space

[ysk24ok]

trafficstars

Environment

• KFP version:

• KFP SDK version: 1.8.13

• All dependencies version:

kfp                      1.8.13
kfp-pipeline-spec        0.1.16
kfp-server-api           1.8.3

Steps to reproduce

import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component
from kfp.v2.dsl import Artifact, Output


@component
def file_generator(path: Output[Artifact]):
    with open(path, "w") as f:
        f.write("Hello World!")

input_path_op = kfp.components.load_component_from_text("""
name: Input path consumer
inputs:
- {name: input path}
implementation:
  container:
    image: alpine
    command:
    - sh
    - -exc
    - cat $0
    - inputPath: input path
""")

output_path_op = kfp.components.load_component_from_text("""
name: output path generator
outputs:
- {name: output path}
implementation:
  container:
    image: alpine
    command:
    - sh
    - -exc
    - echo Hello World! > $0
    - outputPath: output path
""")


@dsl.pipeline(name='my-pipeline')
def my_pipeline():
    file_generator_task = file_generator()
    input_path_op(file_generator_task.outputs["path"])
    # Comment this line out when you want to test the behavior of a component with outputs
    # output_path_op()


kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
    pipeline_func=my_pipeline,
    package_path='pipeline.yaml')

---

Run the code and we will get the following error.

% python3 pipeline.py
/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py:79: UserWarning: V2_COMPATIBLE execution mode is at Beta quality. Some pipeline features may not work as expected.
  warnings.warn('V2_COMPATIBLE execution mode is at Beta quality.'
Traceback (most recent call last):
  File "/Users/y-nishioka/repos/test_kfp/pipeline.py", line 48, in <module>
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1175, in compile
    self._create_and_write_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow
    workflow = self._create_workflow(pipeline_func, pipeline_name,
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1058, in _create_workflow
    workflow = self._create_pipeline_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 786, in _create_pipeline_workflow
    templates = self._create_dag_templates(pipeline, op_transformers)
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 751, in _create_dag_templates
    v2_compat.update_op(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/v2_compat.py", line 158, in update_op
    "metadataPath": op.input_artifact_paths[artifact_name],
KeyError: 'input path'

op.input_artifact_paths is {'input-path': '/tmp/inputs/input_path/data'} and artifact_name is input path so we're getting this error.

---

In the case of output_path_op, we will get the following error.

% python3 pipeline.py
/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py:79: UserWarning: V2_COMPATIBLE execution mode is at Beta quality. Some pipeline features may not work as exp
ected.
  warnings.warn('V2_COMPATIBLE execution mode is at Beta quality.'
Traceback (most recent call last):
  File "/Users/y-nishioka/repos/test_kfp/pipeline.py", line 48, in <module>
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1175, in compile
    self._create_and_write_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow
    workflow = self._create_workflow(pipeline_func, pipeline_name,
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 1058, in _create_workflow
    workflow = self._create_pipeline_workflow(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 786, in _create_pipeline_workflow
    templates = self._create_dag_templates(pipeline, op_transformers)
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/compiler.py", line 751, in _create_dag_templates
    v2_compat.update_op(
  File "/Users/y-nishioka/repos/test_kfp/venv/lib/python3.9/site-packages/kfp/compiler/v2_compat.py", line 187, in update_op
    "metadataPath": op.file_outputs[artifact_name],
KeyError: 'output path'

op.file_outputs is {'output-path': '/tmp/outputs/output_path/data'} and artifact_name is output path so we're getting this error.

Expected result

Compilation succeeds.

v2 SDK doc says

Input and output names are converted to Pythonic names (spaces and symbols are replaced with underscores and letters are converted to lowercase). For example, an input named Input 1 is converted to input_1.

So using a name with empty spaces should be allowed in v2.

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[connor-mccarthy]

Hi @ysk24ok. We no longer support v2-compatible mode. If you'd like to use this compilation style, please consider using the official KFP v2 pre-release (pip install kfp --pre). Otherwise, please use a name without a space for v2 compatible mode.

[AlexandreBrown]

please use a name without a space for v2 compatible mode.

@connor-mccarthy This is not always possible, looking at the katib component, it currently does not work for V2 Compatible mode due to spaces in the input/outputs https://github.com/kubeflow/pipelines/blob/b9a052319e31385f90eb6c8704474376856d3900/components/kubeflow/katib-launcher/component.yaml#L21

In our case we're using Kubeflow 1.4.1, V2 engine is not supported and we don't want to use V1 Legacy mode so we're sort of stuck in the middle and have to use V2 Compatible Mode.

We'd love to upgrade KFP to 2.0 beta but it looks like Instructions on how to upgrade KFP backend for existing Kubeflow installation are Google Cloud Platform specific.
We're not on GCP and for everyone not on GCP (eg: AWS or Azure etc), we're sort of left with no solution.

Ideally there should be clear instructions available for any installation, otherwise it makes everyone not on GCP in a sticky situation IMO.

[connor-mccarthy]

Thanks for explaining @AlexandreBrown. I'm reopening and labelling this as a KFP SDK feature request to handle the spaces issue.

@zijianjoy, do you have any more general KFP BE upgrade instructions that you can provide to @AlexandreBrown?

[zijianjoy]

Hello @AlexandreBrown , can you share more about what your kubernetes environment is (tool and k8s version) and which cloud provider you are using?

In general, you can follow https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#upgrading-kubeflow-pipelines which is platform agnostic. But because you are using full Kubeflow, you need to review the difference between new version and existing version of KFP before upgrading KFP in full kubeflow.

Finally, for any upgrade instruction for full Kubeflow, you can issue request in https://github.com/kubeflow/manifests#kubeflow-manifests. And feel free to contribute writing upgrade instruction too!

[AlexandreBrown]

Hi @zijianjoy , we're using Kubernetes 1.21, running Kubeflow 1.4.1 on AWS (EKS), I believe this version of Kubeflow came with KFP backend 1.8.2, here is the dependencies we have in our notebooks atm :

@surajkota Can you comment on whether it is safe to follow the standalone-deployment instructions to upgrade KFP for Kubeflow deployed on AWS (using RDS-S3-Cognito integrations) ?

[zijianjoy]

@AlexandreBrown For AWS, this is how you deploy vanilla Kubeflow on EKS https://awslabs.github.io/kubeflow-manifests/docs/deployment/vanilla/guide/#kubeflow-pipelines, so you just need to change the version to KFP 1.8.3 and compare the diff. If you are using AWS distribution, then @surajkota can provide more info about it.

Note that current Kubeflow 1.4.1 won't work on Kubernetes 1.22+. Once Kubeflow 1.6 is released, it will be able to support Kubernetes 1.22+

[surajkota]

@AlexandreBrown no, you cannot use standalone deployment instructions to upgrade KFP when using Kubeflow. KFP deployed with a Kubeflow installation is multi user flavour which depends on other components like Istio, cert-manager etc. Kubeflow-1.4 installs KFP-1.7. @zijianjoy to use v2 and address this issue, user would have to install v2.0-alpha correct?

Unfortunately as of today I dont think there is clear guidance on Kubeflow upgrade or if you can just upgrade KFP component in particular Kubeflow versions

[zijianjoy]

@surajkota That is right, for using V2 feature, you need to start using KFP v2.0.0-alpha or above.

In general, you maintain a copy of Kubeflow manifests locally, whenever you want to upgrade a kubeflow component, like KFP. You can compare the diff between old version and new version, and use tools like kpt, helm chart, etc. to perform upgrade.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.