Allow templates to be URLs, like Kubeflow

[alexec]

Summary

Allow resolution of templates by using URLs.

Motivation

So we can have templates in a single place.

Proposal

This is a feature that Kubeflow has and @Ark-kun mentions in #2793.

@mukulikak @jessesuen

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[Ark-kun]

Note that the Kubeflow Pipelines team does not require this feature to be present in Argo (KFP works fine as it is). But this might be a neat feature for people using raw Argo.

Here is an example of using this feature in KF Pipelines: https://github.com/kubeflow/pipelines/blob/master/sdk/python/tests/components/test_data/retail_product_stockout_prediction_pipeline.component.yaml

Here is an example of graph (DAG) using task that references a component (template) using an HTTPS URL.

...
implementation:
  graph:
    tasks:
      Automl split dataset table column names:
        componentRef:
          url: https://raw.githubusercontent.com/kubeflow/pipelines/b3179d86b239a08bf4884b50dbf3a9151da96d66/components/gcp/automl/split_dataset_table_column_names/component.yaml
        arguments:
          dataset_path:
            taskOutput:
              outputName: dataset_path
              taskId: Automl import data from bigquery
          target_column_name:
            graphInput:
              inputName: target_column_name
          table_index: '0'

Here is the schema of the ComponentReference used by KFP components.

[alexec]

Thank you @Ark-kun. I've created this issue and we'll see how many up-votes it get. I can imagine people might like this - I'm intrigued how we can verify URLs.

[merlintang]

Plus one for this. We also have a similar issue. https://github.com/argoproj/argo/issues/2900

[alexec]

We could maybe easily address this using the artifact code we already have.

[merlintang]

@alexec can you give more hints on using the artifact code? do you mean that we can use the executor of the artifact to support URL loading? thanks

[alexec]

No. The real problem here is not rendering templates, but accessing the source in a secure and configurable manner. As we already have code for reading and writing artifacts to various locations - we could reuse that code here.

[merlintang]

Got you, I can look at this. Thanks Alex.

On Mon, May 4, 2020 at 6:45 PM Alex Collins [email protected] wrote:

No. The real problem here is not rendering templates, but accessing the source in a secure and configurable manner. As we already have code for reading and writing artifacts to various locations - we could reuse that code here.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo/issues/2847#issuecomment-623794599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK5R6O4WLDASCUOIK3T3NLRP5VURANCNFSM4MRLZ2JA .