azure-sdk-for-python icon indicating copy to clipboard operation
azure-sdk-for-python copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Sweep component migration is painful

Open fuhuifang opened this issue 2 years ago • 3 comments

Copied from https://msdata.visualstudio.com/Vienna/_workitems/edit/2138112 set sweep setting's complex in 2.0 compared with 1.5.

Previously in v1.5, user can leverage a spec.yaml to configure all the hyperparameters: https://msdata.visualstudio.com/Vienna/_git/aml-ds?path=/recipes/signed-components/canary-sweep-component-mnist/spec.yaml

In 2.0, user needs to manually configure everything in the code: image

This is painful because:

  1. It requires a lot of coding to migrate, compared to other component types.
  2. This makes collaboration difficult, where people want to use different params.
  3. The python script is not as neat as before.

fyi @jfomhover @clthorre @D-W-

fuhuifang avatar Dec 29 '22 19:12 fuhuifang

Hi @fuhuifang, thank you for opening an issue! Is this feedback for a particular package in the Azure SDK, or is this regarding DevOps tooling?

mccoyp avatar Dec 29 '22 19:12 mccoyp

Hi @mccoyp, it’s about a component type in SDK V2. Here’s an example https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/pipeline_with_hyperparameter_sweep, but we submit the job via SDK instead of cli + the yaml definition.

fuhuifang avatar Dec 30 '22 18:12 fuhuifang

Okay, thank you for the details! I'll tag some folks from the ML team who may be able to help -- we'll get back to you as soon as possible. @luigiw @azureml-github

mccoyp avatar Dec 30 '22 18:12 mccoyp

I synced up with Fukui offline. The problem here seems to be the lack of sweep component support.

@wangchao1230, is it possible to allow customers to load a sweep job from a Yaml file and use the sweep job in a pipeline job?

luigiw avatar Jan 11 '23 21:01 luigiw

@luigiw @fuhuifang the capability we provide for SDK and CLI is parity. You can find related yaml sample here: https://github.com/Azure/azureml-examples/blob/main/cli/jobs/pipelines-with-components/pipeline_with_hyperparameter_sweep/pipeline.yml

cloga avatar Jan 12 '23 05:01 cloga

Thank you for sharing the example, @cloga. Do you also have a DSL example, where a sweep job object is used in a pipeline dsl function?

luigiw avatar Jan 13 '23 00:01 luigiw

@luigiw I don't think we want customer to use sweep job object in SDK, instead we recommend customer to use builder style on command component command_component.sweep().

Pipeline DSL: https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1c_pipeline_with_hyperparameter_sweep/pipeline_with_hyperparameter_sweep.ipynb

Standalone sweep job: https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/single-step/scikit-learn/train-hyperparameter-tune-deploy-with-sklearn/train-hyperparameter-tune-with-sklearn.ipynb

cloga avatar Feb 27 '23 02:02 cloga