AML Pipeline: azureml:dataset_name:@latest does not return latest registered dataset

[thegitofdaniel]

I have submited a pipeline run using yaml + CLI2 command "az ml job create -f filename".

The pipeline gets created. However, the wrong version of dataset is selected if I use "azureml:dataset_name:@latest".

In the yaml I declare:

inputs:
  training_data: 
      type: mltable 
      path: azureml:greenfield_slp_combined_input@latest

I should get version 5 of the dataset, instead the pipeline gets version 1.

Not being able to get the latest version of the dataset is quite a big issue.

On a separate topic, "@latest" works fine when I try to select the latest environment.

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: c56bf00c-ec47-4a12-9a87-137fb29893e0
• Version Independent ID: 86302e6a-7281-fb13-3702-63ea37622606
• Content: Access data in a job - Azure Machine Learning
• Content Source: articles/machine-learning/how-to-read-write-data-v2.md
• Service: machine-learning
• Sub-service: mldata
• GitHub Login: @ynpandey
• Microsoft Alias: yogipandey

[RamanathanChinnappan-MSFT]

@thegitofdaniel

Thanks for your feedback! We will investigate and update as appropriate.

[Naveenommi-MSFT]

Hi @thegitofdaniel We have assigned this to content author @ynpandey to check and share his valuable insights on this.

[thegitofdaniel]

Please notice: it worked when I had a V2 datasets. It didn't get the latest only when I had a V1 dataset. Backward compatibility should be guaranteed.

[SturgeonMi]

Thank you Daniel for the feedback! It's a known issue and we are currently working on it. This issue should be resolved by end of June.