Conditional Pipeline Job steps Azure ML v2

[tommyJimmy87]

What example? Describe it

How to run steps conditionally from a Job pipeline (yaml) if possible at all.

Example: I want to run a step (fine-tuning for example) only on some boolean condition and I would like to pass this condition inside the yaml file which describe the pipeline Job. ( In Kubeflow this is possible through dsl.condition in the python sdk )

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
experiment_name: exp
description: description

jobs:
  #Training step
  train:
    name: training
    display_name: Model-training
    environment: azureml:ccep_train_env_docker@latest
    code: ../../training
    command: >-
      python train_aml.py
**IF <some condition>:**
  #Fine tune step
  finetuning:
    name: fine_tuning
    display_name: Fine-Tuning-Model
    code: ../../finetuning
    command: >-
      python finetune_aml.py

Not sure this is possible at all, in case would be great to have an ehnancement, is super useful.

Thanks

[cloga]

Hi @tommyJimmy87 ,

Thanks for your feedback, we do have initial thinking about control flow, is it possible to find some time to discuss more detail about your scenario?

[magiob]

A similar setup to Github workflows or Azure devops pipelines, would be very useful here. Elaborating on @tommyJimmy87 answer:

jobs:
  #Training step
  train:
    name: training
    display_name: Model-training
    environment: azureml:ccep_train_env_docker@latest
    code: ../../training
    outputs:
      score:
        type: number
    command: >-
      python train_aml.py

  #Fine tune step
  finetuning:
    **if: jobs.train.outputs.score > 0.5**
    name: fine_tuning
    display_name: Fine-Tuning-Model
    code: ../../finetuning
    command: >-
      python finetune_aml.py

[tdegamboda]

Are there any updates on this? I too would ideally like to control the triggering of components in a pipeline using an if else condition. At the moment I imagine it would sit well in the pipeline definition/pipeline YAML file, but I would really appreciate any updates on this matter.

[dazeandconfusion]

Any updates on the same topic but using Python SDK for conditions in pipelines?

[brn7rng]

Are there any updates here? I would also profit from this feature.

[zsuhor]

We would also like to have this feature. :)

[ammaradam]

This feature would be useful for me as well. avoids unnecessary burden running a job just to check a condition. and also gives a cleaner workflow

[biarne-a]

I am currently migrating my pipeline from google cloud (Vertex AI pipeline) to Azure ML and I am pretty astonished that this feature is not available in Azure... Especially since it is also available in the open source version (eg. Kubeflow)