[BUG] DataLake Management: Default API Version is not recognized

[BowenYang666]

trafficstars

Library name and version

Microsoft.Azure.Management.DataLake.Analytics 3.0.0

Describe the bug

I got below exception when I tried to submit a job to Azure Data Lake Analytics using .NET SDK

my code is like this: var res = adlaJobClient.Job.Get(adlaAccountName, new Guid("cc75454e-5ab0-4cc0-84f0-35bd7b09b8a2"));

error msg: Unhandled Exception: Microsoft.Rest.Azure.CloudException: api-version '2016-11-01' is invalid. Valid api-version: '2018-02-01-preview', '2017-09-01-preview' at Microsoft.Azure.Management.DataLake.Analytics.JobOperations.<GetWithHttpMessagesAsync>d__10.MoveNext()

Not just Job.Get method get error msg, I've tried other methods provided in the microsoft doc, there all have same error.

I used these packages and wrote the code as suggested in the microsoft doc: https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-manage-use-dotnet-sdk

Install-Package -Id Microsoft.Rest.ClientRuntime.Azure.Authentication -Version 2.3.1 Install-Package -Id Microsoft.Azure.Management.DataLake.Analytics -Version 3.0.0 Install-Package -Id Microsoft.Azure.Management.DataLake.Store -Version 2.2.0 Install-Package -Id Microsoft.Azure.Management.ResourceManager -Version 1.6.0-preview Install-Package -Id Microsoft.Azure.Graph.RBAC -Version 3.4.0-preview

I tried to upgrade the Microsoft.Azure.Management.DataLake.Analytics to latest version, but still didn't work. So I wonder if there is restrictions in using this sdk.

Expected behavior

adla sdk can be invoked properly.

Actual behavior

got error showing api-version is not valid when using adla sdk:

Unhandled Exception: Microsoft.Rest.Azure.CloudException: api-version '2016-11-01' is invalid. Valid api-version: '2018-02-01-preview', '2017-09-01-preview' at Microsoft.Azure.Management.DataLake.Analytics.JobOperations.<GetWithHttpMessagesAsync>d__10.MoveNext()

Reproduction Steps

`using System; using System.IO; using System.Threading; using System.Security.Cryptography.X509Certificates;

using Microsoft.Rest; using Microsoft.Rest.Azure.Authentication; using Microsoft.Azure.Management.DataLake.Analytics; using Microsoft.Azure.Management.DataLake.Analytics.Models; using Microsoft.Azure.Management.DataLake.Store; using Microsoft.Azure.Management.DataLake.Store.Models; using Microsoft.IdentityModel.Clients.ActiveDirectory; using Microsoft.Azure.Graph.RBAC; using Microsoft.Azure.Management.DataLake.Analytics.Models; using Microsoft.Rest.Azure.OData;

namespace ADLATest {

public static class Program
{
    public static string TENANT = "microsoft.onmicrosoft.com";
    public static string CLIENTID = "1950a258-227b-4e31-a9cf-717495945fc2";
    public static System.Uri ARM_TOKEN_AUDIENCE = new System.Uri(@"https://management.core.windows.net/");
    public static System.Uri ADL_TOKEN_AUDIENCE = new System.Uri(@"https://datalake.azure.net/");
    public static System.Uri GRAPH_TOKEN_AUDIENCE = new System.Uri(@"https://graph.windows.net/");

    static void Main(string[] args)
    {
        string adlaAccountName = "ffo-prod-c14";
        string clientId = "xxxx";
        string secretKey = "xxxx";

        string domain = "cdc5aeea-15c5-4db6-b079-fcadd2505dc2";
        var adlTokenAudience = new Uri(@"https://datalake.azure.net/");

        var adlCreds = GetCredsServicePrincipalSecretKey(domain, adlTokenAudience, clientId, secretKey);

        var adlaJobClient = new DataLakeAnalyticsJobManagementClient(adlCreds);

        var odq = new ODataQuery<JobInformation> { Filter = "result eq 'Failed'" };
        var jobs = adlaJobClient.Job.List(adlaAccountName, odq);
        foreach (var j in jobs)
        {
            Console.WriteLine($"{j.Name}\t{j.JobId}\t{j.Type}\t{j.StartTime}\t{j.EndTime}");
        }
    }

    private static ServiceClientCredentials GetCredsServicePrincipalSecretKey(string domain, Uri tokenAudience, string clientId, string secretKey)
    {
        SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

        var serviceSettings = ActiveDirectoryServiceSettings.Azure;
        serviceSettings.TokenAudience = tokenAudience;

        var creds = ApplicationTokenProvider.LoginSilentAsync(domain, clientId, secretKey, serviceSettings).GetAwaiter().GetResult();

        return creds;
    }
}

} ` run the code with these versions, you will get error. Install-Package -Id Microsoft.Rest.ClientRuntime.Azure.Authentication -Version 2.3.1 Install-Package -Id Microsoft.Azure.Management.DataLake.Analytics -Version 3.0.0 Install-Package -Id Microsoft.Azure.Management.DataLake.Store -Version 2.2.0 Install-Package -Id Microsoft.Azure.Management.ResourceManager -Version 1.6.0-preview Install-Package -Id Microsoft.Azure.Graph.RBAC -Version 3.4.0-preview

Environment

.NET SDK (reflecting any global.json): Version: 6.0.301 Commit: 43f9b18481

Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.301\

Host (useful for support): Version: 6.0.6 Commit: 7cca709db2

.NET SDKs installed: 3.1.420 [C:\Program Files\dotnet\sdk] 5.0.408 [C:\Program Files\dotnet\sdk] 5.0.409 [C:\Program Files\dotnet\sdk] 6.0.100 [C:\Program Files\dotnet\sdk] 6.0.106 [C:\Program Files\dotnet\sdk] 6.0.202 [C:\Program Files\dotnet\sdk] 6.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

IDE: Microsoft Visual Studio Enterprise 2022 (64-bit) - Current Version 17.2.4

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Service Bus:0.07679948,Data Lake Storage Gen2:0.07147485,ARM:0.06971483'

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Data Lake Storage Gen2:0.08268967,Service Bus:0.07306276,ARM:0.06871974'

[jteves]

Getting this exception in my program, too:

Microsoft.Rest.Azure.CloudException
  HResult=0x80131500
  Message=api-version '2016-11-01' is invalid. Valid api-version: '2018-02-01-preview', '2017-09-01-preview'
  Source=Microsoft.Azure.Management.DataLake.Analytics
  StackTrace:
   at Microsoft.Azure.Management.DataLake.Analytics.PipelineOperations.<ListWithHttpMessagesAsync>d__5.MoveNext()
   at Microsoft.Azure.Management.DataLake.Analytics.PipelineOperationsExtensions.<ListAsync>d__1.MoveNext()
   at Microsoft.Azure.Management.DataLake.Analytics.PipelineOperationsExtensions.List(IPipelineOperations operations, String accountName, Nullable`1 startDateTime, Nullable`1 endDateTime)

Is there a valid version of Microsoft.Azure.Management.DataLake.Analytics that the PipelineOperationsExtensions.List call would work for (reference link)?

Lastly I did find a reference for the api-version here:

                new Tuple<string, string, string>("DataLakeAnalytics", "Accounts", "2016-11-01"),

And here:

            ApiVersion = "2016-11-01";

Can these simply point to valid api-versions, 2018-02-01-preview or 2017-09-01-preview?

[jteves]

Update: I was able to get around this error by using the 3.5.3-preview version of the Microsoft.Azure.Management.DataLake.Analytics package.

[dvbb]

Microsoft.Azure.Management.DataLake.Analytics has been deprecated. We suggest migrating to the new SDK Azure.ResourceManager.DataLakeAnalytics that can specify the API version.

Here is a sample

using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.Resources;
using Azure.ResourceManager.DataLakeAnalytics;

// Create ArmClient 
ClientSecretCredential clientSecretCredential = new ClientSecretCredential(tenantId, clientId, clientSecret);
var options = new ArmClientOptions();
options.SetApiVersion("microsoft.resources/resourceGroups", "2021-01-01");
options.SetApiVersion("microsoft.DataLakeAnalytics/accounts", "2018-02-01-preview");
ArmClient armClient = new ArmClient(clientSecretCredential, "subscriptionId", options);

// Get a resource group
var rg = await (armClient.GetDefaultSubscriptionAsync().Result.GetResourceGroups()).GetAsync("resource group name");

// Get DataLake Analytics Account
var adla = await rg.Value.GetDataLakeAnalyticsAccounts().GetAsync("adlaName");

[github-actions[bot]]

Hi @BowenYang666. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @BowenYang666, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!