elyra icon indicating copy to clipboard operation
elyra copied to clipboard

allow setting component inputs from files (KFP & Airflow)

Open thesuperzapper opened this issue 2 years ago • 2 comments

Currently, we only provide two methods to specify component inputs:

  1. Plain Strings (no newlines)
  2. From the output of a parent node

We can provide a third option that selects a local file as the input. During pipeline run/export we would pass the content of this file as a string to the input.

For example, if a component takes an SQL query as an input, it would be amazing to just reference a local query.sql file, as right now newlines can't even be input:

Screen Shot 2022-06-02 at 16 05 47


Considerations:

  • KFP generates Argo-Workflow YAML, and there are limits on how big of a YAML Kubenrentes will accept, so we might want to fail if the input file would cause the YAML to be too big (but I am not sure on the exact max value).

thesuperzapper avatar Jun 02 '22 06:06 thesuperzapper

This would be similar to the 'local file dependencies' we currently support for generic components, right? For now users could use notebooks/scripts to accomplish this, instead of custom components. (Not arguing against it, just wanted to bring this up for others who come across this request and are looking for a solution that is available today.)

ptitzler avatar Jun 02 '22 14:06 ptitzler

This would be similar to the 'local file dependencies' we currently support for generic components, right? For now, users could use notebooks/scripts to accomplish this, instead of custom components. (Not arguing against it, just wanted to bring this up for others who come across this request and are looking for a solution that is available today.)

@ptitzler the main difference to the "local file dependencies" is that this approach would not use the COS bucket, as the content of the input file would be read at pipeline "run" and "export" time and then input as a string to the Kubeflow/Airflow component.

thesuperzapper avatar Jun 02 '22 23:06 thesuperzapper

Since pipelines cannot be exported or run without these input files being present, the elyra-pipeline describe command also needs to be extended to list them as dependencies.

ptitzler avatar Sep 07 '22 09:09 ptitzler