elyra
elyra copied to clipboard
allow setting component inputs from files (KFP & Airflow)
Currently, we only provide two methods to specify component inputs:
- Plain Strings (no newlines)
- From the output of a parent node
We can provide a third option that selects a local file as the input. During pipeline run/export we would pass the content of this file as a string to the input.
For example, if a component takes an SQL query as an input, it would be amazing to just reference a local query.sql
file, as right now newlines can't even be input:
Considerations:
- KFP generates Argo-Workflow YAML, and there are limits on how big of a YAML Kubenrentes will accept, so we might want to fail if the input file would cause the YAML to be too big (but I am not sure on the exact max value).
This would be similar to the 'local file dependencies' we currently support for generic components, right? For now users could use notebooks/scripts to accomplish this, instead of custom components. (Not arguing against it, just wanted to bring this up for others who come across this request and are looking for a solution that is available today.)
This would be similar to the 'local file dependencies' we currently support for generic components, right? For now, users could use notebooks/scripts to accomplish this, instead of custom components. (Not arguing against it, just wanted to bring this up for others who come across this request and are looking for a solution that is available today.)
@ptitzler the main difference to the "local file dependencies" is that this approach would not use the COS bucket, as the content of the input file would be read at pipeline "run" and "export" time and then input as a string to the Kubeflow/Airflow component.
Since pipelines cannot be exported or run without these input files being present, the elyra-pipeline describe
command also needs to be extended to list them as dependencies.