Add initial support for rjsf in pipeline properties

[marthacryan]

Very rough draft - corresponding to elyra-ai/pipeline-editor#197. Currently allows for editing and saving the pipeline properties and the generic node properties but not most of the custom component node properties (mostly due to needing to implement the input/ output paths).

Todo:

• [x] basic pipeline properties support
• [x] basic generic node properties support
• [x] basic "canvas properties" support
• [ ] finish airflow support
• [x] link in custom components
• [x] link in validation
• [x] add in required fields
• [x] add inputpath as oneof
• [ ] pipeline migration
• [ ] go through code to make sure all utilities and edits etc. are all corresponding to the updated schema

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot[bot]]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[akchinSTC]

@ajbozarth - Do we think this will make the 3.11 milestone this friday?

[ptitzler]

Not specific to RJSF migration but since the PR already modifies canvas_properties_template.jinja2, can we add descriptions for the inputs and outputs headers like we already have for additional properties?

[akchinSTC]

When using the papermill operator, the area for the notebook parameter in the component properties side panel isnot rendering, but is a required field.

[kiersten-stokes]

When using the papermill operator, the area for the notebook parameter in the component properties side panel isnot rendering, but is a required field.

I think I found a solution to this! Should be working now

[ptitzler]

Re-assigned to 3.12 milestone since the enhancements that users would benefit from (e.g. https://github.com/elyra-ai/elyra/issues/2738, https://github.com/elyra-ai/elyra/issues/2750) won't be ready before cut-off according to https://github.com/elyra-ai/elyra/issues/2738#issuecomment-1219676250

[ptitzler]

A few things I didn't see listed in the todo list in the official list.

Pipeline properties

• Heading font size is smaller than text font size (see "Pipeline Runtime" vs "Kubeflow Pipelines")

• Sections are not clearly separated (compare this to the layout used for node properties (shown below), which is somewhat better due to indentation).

  Layout for node properties

Since descriptions (for headings/properties) are now only displayed as tool tips, they are much harder to read if they are longer than a few words. In many cases a new user will likely require additional information, which we make available in the documentation. How can we make links available? Could the (?) icons perhaps serve two purposes: as a visual indication that there is info available (via tool tip) and as an HREF to a URL that would open up in a new browser window if one clicks on the icon.

[ajbozarth]

@kiersten-stokes I pushed the fix to migration.

I also updated some of the snapshot files, one of them will need to be regenerated again after we fix the issue with persisting the schema.

As for the complex pipeline test, that one uses accessors that no longer exist to fill out the form, I tried to update it to use the new accessors, but there are none in the new form. @marthacryan

[marthacryan]

• Heading font size is smaller than text font size (see "Pipeline Runtime" vs "Kubeflow Pipelines")

Fixed

• Sections are not clearly separated (compare this to the layout used for node properties (shown below), which is somewhat better due to indentation).

Fixed

Since descriptions (for headings/properties) are now only displayed as tool tips, they are much harder to read if they are longer than a few words. In many cases a new user will likely require additional information, which we make available in the documentation. How can we make links available? Could the (?) icons perhaps serve two purposes: as a visual indication that there is info available (via tool tip) and as an HREF to a URL that would open up in a new browser window if one clicks on the icon.

I can add this as a todo, I'd need to discuss this a little further

[ptitzler]

• Heading font size is smaller than text font size (see "Pipeline Runtime" vs "Kubeflow Pipelines")

Fixed

Heading font size is still 12px vs text font size of 16px

---

[generic node] adjacent buttons should not touch

---

tool tips for custom components aren't displayed properly

---

combination of several issues if one doesn't enter a text value in list-valued input:

• set this as pipeline default

  

• non-existent value rendered in node properties as pipeline default

• stored as null in pipeline file

  

[ptitzler]

@kiersten-stokes is this ready for testing or still in wip? https://github.com/elyra-ai/elyra/pull/2767 indicates that this PR will take care of it

[kiersten-stokes]

is this ready for testing or still in wip? #2767 indicates that this PR will take care of it

@ptitzler yes, this is ready to test! I saw your comment on #2759 as well and added the relevant logic to the describe command.

[marthacryan]

Heading font size is still 12px vs text font size of 16px

I think I hadn't updated the link with that fix - it looks right on my end but let me know if you're still seeing it on a more recent build.

[generic node] adjacent buttons should not touch

Fixed

tool tips for custom components aren't displayed properly

Fixed

• non-existent value rendered in node properties as pipeline default

Fixed

[ptitzler]

make install in a clean environment fails due to linting issues:

yarn eslint --max-warnings=0
yarn run v1.22.4
$ eslint . --fix --ignore-path .gitignore --ext .ts,.tsx,.js --max-warnings=0

/Users/patti/elyra_dev2/martha-rjsf/elyra/packages/pipeline-editor/src/PipelineEditorWidget.tsx
  306:28  warning  Missing return type on function                                                                                        @typescript-eslint/explicit-function-return-type
  338:6   warning  React Hook useCallback has a missing dependency: 'removeNullValues'. Either include it or remove the dependency array  react-hooks/exhaustive-deps

✖ 2 problems (0 errors, 2 warnings)

ESLint found too many warnings (maximum: 0).
error Command failed with exit code 1.

[ptitzler]

@kiersten-stokes: overall file input looks good but I managed to break this scenario:

• choose an input property that has a default value defined (e.g. for bashoperator use output_encoding, which is set to utf-8)
• change the selection from ... a string value to ... select a file
• the default value is now interpreted as a file name, which it isn't

the incorrect values also surface in the output of the elyra-pipeline describe command:

Local file dependencies:
    - {}
    - utf-8
    - command.txt
    - true-or-false.txt

[ptitzler]

tool tips for custom components aren't displayed properly

Fixed

@marthacryan now no properties are displayed

main branch output:

[ptitzler]

It appears that we are now persisting a property named component_source_header in the .pipeline file:

            },
            "component_source_header": null
          },

This property is only used as a visualization aid ("section header") in the UI and shouldn't be persisted:

From https://github.com/marthacryan/elyra/blob/pipeline-rjsf/elyra/templates/components/canvas_properties_template.jinja2#L123-L129

"component_source_header": {
      "type": "null",
      "title": "Component Source",
      "uihints": {
        "ui:field": "header"
      }
    },

[kiersten-stokes]

choose an input property that has a default value defined (e.g. for bashoperator use output_encoding, which is set to utf-8) change the selection from ... a string value to ... select a file the default value is now interpreted as a file name, which it isn't

@ptitzler yeah, just checked and I see this too. I think the fix will be in the frontend, but I'll double check at the meeting this AM!

[marthacryan]

@ptitzler Thanks for checking it out again! I fixed the lint and the extra null value. I'm working on the airflow components tooltips issue

[ptitzler]

@ajbozarth migration of the noaa tutorial pipeline fails:

[ptitzler]

The validation message is invalid if one does not specify a file as string input:

Note the value of value, which is not what the user provided as input. The expected value would be null (undefined).

[ptitzler]

Tooltip for custom nodes in Airflow are now looking good ...

For property values that are provided via input files (URL in node download file in the example below), can we prepend the file name with a string that indicates that we are displaying a reference, not the value? The count rows node is an example of a property that is also defined via reference, specifically a reference to the output of an upstream node:

[kiersten-stokes]

The validation message is invalid if one does not specify a file as string input:

Good point! I've fixed validation to allow an empty string for file input types for non-required properties. From here, the processor behaves the same way as an empty string for raw input as well.

[ptitzler]

The validation message is invalid if one does not specify a file as string input:

Good point! I've fixed validation to allow an empty string for file input types for non-required properties. From here, the processor behaves the same way as an empty string for raw input as well.

Thank you! Looking good. It appears to preserve default values, which I would expect to happen since the user didn't provide a property value.

[kevin-bates]

Congratulations (and thank you :heart:) to everyone that worked on this - it's one for the ages! :tada:

[thesuperzapper]

I would also love to thank everyone for their amazing effort!!!

❤️