azureml-examples icon indicating copy to clipboard operation
azureml-examples copied to clipboard

Provide example of best practice for passing objects (e.g. azureml.core.Dataset) between custom components in pipelines

Open caitriggs opened this issue 2 years ago • 0 comments

What example? Describe it

In the AMLS Designer it appears that components can have a Dataset output type. What's the equivalent to be able to pass a Dataset object from the output of a custom Component A to the input of custom Component B? The only examples I see about dealing with inputs/outputs of custom components is with writing out data to paths and using the path as an input to the next component.

I guess this is simply a limitation of custom components vs components in Designer? That custom components by nature are registered separately from the pipeline run they are composed within, so there's no way to configure passing objects between them, since their scripts are registered separately, other than writing out a pickle file and picking it up in the next component via the output path of it's component? Since Designer is ran from the AMLS workspace runtime, this allows for Dataset objects to be passed between?

So is the best practice for passing Dataset objects (and other objects) between AML Custom Components within an AML Pipeline via setting an output path for a pickle file?

example of output as Dataset object in Designer

caitriggs avatar Mar 16 '22 00:03 caitriggs