elyra icon indicating copy to clipboard operation
elyra copied to clipboard

elyra-pipeline validate CLI command fails for pipelines with parameters and default properties

Open mamurak opened this issue 1 year ago • 1 comments

Describe the issue Running elyra-pipeline validate and by extension elyra-pipeline export fails silently with pipelines that contain both pipeline parameters and default properties.

To Reproduce Steps to reproduce the behavior:

  1. Create a simple Elyra pipeline with a single node.
  2. Add a pipeline parameter in the Pipeline Parameters configuration tab.
  3. Open the Pipeline Properties configuration tab.
  4. Save the pipeline, e.g. as test.pipeline.
  5. Validate the pipeline through the CLI: elyra-pipeline validate test.pipeline.

Screenshots or log output The following error message without details is printed after running the above validation command:

Log Output
(app-root) (app-root) elyra-pipeline validate test.pipeline 

──────────────────────────────────────────────────────────────── Elyra Pipeline Validation ────────────────────────────────────────────────────────────────

Validating pipeline... Error: Pipeline validation FAILED.

Expected behavior Given that pipelines with parameters and default properties are valid in general, the validation command should resolve without any errors. In case of actual issues within the pipeline file, there should be detailed information about these issues.

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.15.0
  • Installation source: quay.io/opendatahub/workbench-images:cuda-jupyter-tensorflow-ubi9-python-3.9
  • Deployment type: Red Hat OpenShift Data Science 1.33.0
  • Operating system: linux

mamurak avatar Oct 13 '23 15:10 mamurak

Example pipeline:

{
  "doc_type": "pipeline",
  "version": "3.0",
  "json_schema": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json",
  "id": "elyra-auto-generated-pipeline",
  "primary_pipeline": "primary",
  "pipelines": [
    {
      "id": "primary",
      "nodes": [
        {
          "id": "54d57acc-4ad0-410e-8d79-268cd52a45e9",
          "type": "execution_node",
          "op": "execute-python-node",
          "app_data": {
            "component_parameters": {
              "pipeline_parameters": [
                "param"
              ],
              "dependencies": [],
              "include_subdirectories": false,
              "outputs": [],
              "env_vars": [],
              "kubernetes_pod_annotations": [],
              "kubernetes_pod_labels": [],
              "kubernetes_secrets": [],
              "kubernetes_shared_mem_size": {},
              "kubernetes_tolerations": [],
              "mounted_volumes": [],
              "filename": "untitled.py",
              "runtime_image": "quay.io/modh/runtime-images@sha256:7dd23e58291cad7a0ab4a8e04bda06492f2c027eb33b226358380db58dcdd60b"
            },
            "label": "",
            "ui_data": {
              "label": "untitled.py",
              "image": "/notebook/fraud-detection/tensorflow/static/elyra/python.svg",
              "x_pos": 505,
              "y_pos": 210,
              "description": "Run Python script"
            }
          },
          "inputs": [
            {
              "id": "inPort",
              "app_data": {
                "ui_data": {
                  "cardinality": {
                    "min": 0,
                    "max": -1
                  },
                  "label": "Input Port"
                }
              }
            }
          ],
          "outputs": [
            {
              "id": "outPort",
              "app_data": {
                "ui_data": {
                  "cardinality": {
                    "min": 0,
                    "max": -1
                  },
                  "label": "Output Port"
                }
              }
            }
          ]
        }
      ],
      "app_data": {
        "ui_data": {
          "comments": []
        },
        "version": 8,
        "runtime_type": "KUBEFLOW_PIPELINES",
        "properties": {
          "name": "untitled",
          "runtime": "Data Science Pipelines",
          "pipeline_parameters": [
            {
              "name": "param",
              "description": "",
              "default_value": {
                "type": "String",
                "value": "val"
              },
              "required": false
            }
          ],
          "pipeline_defaults": {
            "kubernetes_shared_mem_size": {},
            "kubernetes_pod_labels": [],
            "mounted_volumes": [],
            "kubernetes_pod_annotations": [],
            "kubernetes_tolerations": [],
            "kubernetes_secrets": [],
            "env_vars": []
          }
        }
      },
      "runtime_ref": ""
    }
  ],
  "schemas": []
}

Validation through the CLI fails. Remove either pipeline_parameters or pipeline_defaults, and validation succeeds.

mamurak avatar Oct 13 '23 15:10 mamurak