azure-sdk-for-net icon indicating copy to clipboard operation
azure-sdk-for-net copied to clipboard

Published 20 hours ago • verified publisher icon Azure

[BUG] GetServiceBusQueues returns partial list unless throttled

Open bo-ravnborg opened this issue 1 year ago • 1 comments

Library name and version

Azure.ResourceManager.ServiceBus 1.0.0

Describe the bug

I have an Azure Service Bus Namespace with 3,000 queues, and would like to fetch them all using GetServiceBusQueues. If I just try to fetch by doing GetServiceBusQueues().GetAllAsync().ToListAsync() it will return an arbitrary number of queues, and not the full list.

Expected behavior

Return all queues when calling GetServiceBusQueues().GetAllAsync() or if not possible it should throw an exception.

Actual behavior

Returns an incomplete list of queues.

Reproduction Steps

public async Task GetAllQueues()
{
    var armClient = new ArmClient(new DefaultAzureCredential(new DefaultAzureCredentialOptions { TenantId = TenantId }));
    var subscription = await armClient.GetDefaultSubscriptionAsync();
    var resourceGroupResponse = await subscription.GetResourceGroupAsync(ServiceBusResourceGroup);
    var resourceGroup = resourceGroupResponse.Value;
    var namespaceCollection = resourceGroup.GetServiceBusNamespaces();
    var serviceBusNamespace = await namespaceCollection.GetAsync(ServiceBusNamespace); // Existing service bus namespace with 3.000 queues.

    var serviceBusQueues = await serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync().ToListAsync();
    Console.WriteLine($"Count without waiting: {serviceBusQueues.Count}"); // Example run: Count without waiting: 1918
    
    // Wait a minute to reset the throttling
    await Task.Delay(TimeSpan.FromMinutes(1));

    var serviceBusQueues2 = new List<ServiceBusQueueResource>(); 
    await foreach (var queue in serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync())
    {
        serviceBusQueues2.Add(queue);
        await Task.Delay(10); // To avoid throttling
    }
    Console.WriteLine($"Count with waiting: {serviceBusQueues2.Count}"); // Example run: Count with waiting: 3000
}

Environment

.NET SDK Version 7.0.306 Both Windows and Linux has the issues.

bo-ravnborg avatar Aug 02 '23 14:08 bo-ravnborg

Thank you for your feedback. Tagging and routing to the team member best able to assist.

jsquire avatar Aug 02 '23 14:08 jsquire

@bo-ravnborg Regarding the part about ToListAsync(), we are not responsible for this part, so we know very little about its operating mechanism. Therefore, we are sorry that we cannot answer your confusion. If using await foreach (var queue in serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync()) to traverse can achieve your target requirements, then I think the Azure dotnet SDK cannot provide the feature optimization you need for the time being. Thank you for your feedback.

mcgallan avatar Mar 13 '24 08:03 mcgallan

Hi @bo-ravnborg. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

github-actions[bot] avatar Mar 13 '24 08:03 github-actions[bot]

Hi @mcgallan,

Thank you for your suggestion. I've tried implementing the await foreach pattern on serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync(), but this alone doesn't address the underlying issue. Interestingly, incorporating await Task.Delay(10); seems to be the workaround that actually resolves the problem at hand.

This workaround suggests a potential inefficiency or limitation within the API when fetching the full list without manual throttling. Ideally, this kind of rate-limiting or throttling would be handled internally by the library, providing a seamless experience to the end user.

bo-ravnborg avatar Mar 13 '24 09:03 bo-ravnborg

Hi @bo-ravnborg , if you need to set custom throttling policy, there is a work around.

ThrottlingPolicy.cs
using Azure.Core;
using Azure.Core.Pipeline;

namespace MgmtTest
{
    internal class ThrottlingPolicy : HttpPipelineSynchronousPolicy
    {
        private static string REMAININGHEADER = "x-ms-user-quota-remaining";
        private static string REQUESTAFTERHEADER = "x-ms-user-quota-resets-after";
        string _remaining;
        string _resetsAfter;

        internal ThrottlingPolicy(string remaining, string resetsAfter)
        {
            _remaining = remaining;
            _resetsAfter = resetsAfter;
        }
        public override void OnSendingRequest(HttpMessage message)
        {
            if (!message.Request.Headers.TryGetValue(REMAININGHEADER, out _))
            {
                message.Request.Headers.Add(REMAININGHEADER, _remaining);
            }
            if (!message.Request.Headers.TryGetValue(REQUESTAFTERHEADER, out _))
            {
                message.Request.Headers.Add(REMAININGHEADER, _resetsAfter);
            }
        }
    }
}

ArmClientOptions clientOptions = new ArmClientOptions();
ThrottlingPolicy throttlingPolicy = new ThrottlingPolicy("3", "00:00:03");
clientOptions.AddPolicy(throttlingPolicy, HttpPipelinePosition.PerCall);
var client = new ArmClient(new ClientSecretCredential(tenantId, clientId, clientSecret), subscription, clientOptions);
var sub = await client.GetDefaultSubscriptionAsync();
var resourceGroup = (await sub.GetResourceGroups().GetAsync("<Name>")).Value;

HarveyLink avatar Mar 13 '24 09:03 HarveyLink

I just tested this custom ThrottlingPolicy, and it does not seem to make any difference. GetAllAsync still returns somewhat less than 3000 queues when run against a Service Bus namespace containing 3000 queues.

However, I did a test with enumerating the pages in the AsyncPageable directly, and this shows the API responses suddenly start skipping over way too many elements:

var serviceBusQueues = new List<ServiceBusQueueResource>();
await foreach (var page in serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync().AsPages())
{
    Console.WriteLine($"Got page containing {page.Values.Count} values with continuation token {page.ContinuationToken}");
    serviceBusQueues.AddRange(page.Values);
}
Console.WriteLine($"Count with waiting: {serviceBusQueues.Count}");

Got page containing 100 values with continuation token https://management.azure.com/subscriptions/{Subscription}/resourceGroups/{ResourceGroup}/providers/Microsoft.ServiceBus/namespaces/{Namespace}/queues?api-version=2021-11-01&$skip=100&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=200&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=300&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=400&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=500&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=600&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=700&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=800&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=900&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1000&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1100&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1200&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1537&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2010&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2424&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2581&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2700&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2815&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2915&$top=100
Got page containing 85 values with continuation token 
Count with waiting: 1985

I wouldn't even call this a throttling issue per-se and I don't think it's something the SDK can fix. It seems to be a bug in the API endpoint. @HarveyLink, do you agree, and if so, do you know how we could bring this to the attention of the relevant team?

kimsey0 avatar Mar 18 '24 15:03 kimsey0

I just tested this custom ThrottlingPolicy, and it does not seem to make any difference. GetAllAsync still returns somewhat less than 3000 queues when run against a Service Bus namespace containing 3000 queues.

However, I did a test with enumerating the pages in the AsyncPageable directly, and this shows the API responses suddenly start skipping over way too many elements:

var serviceBusQueues = new List<ServiceBusQueueResource>();
await foreach (var page in serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync().AsPages())
{
    Console.WriteLine($"Got page containing {page.Values.Count} values with continuation token {page.ContinuationToken}");
    serviceBusQueues.AddRange(page.Values);
}
Console.WriteLine($"Count with waiting: {serviceBusQueues.Count}");

Got page containing 100 values with continuation token https://management.azure.com/subscriptions/{Subscription}/resourceGroups/{ResourceGroup}/providers/Microsoft.ServiceBus/namespaces/{Namespace}/queues?api-version=2021-11-01&$skip=100&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=200&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=300&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=400&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=500&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=600&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=700&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=800&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=900&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1000&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1100&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1200&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=1537&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2010&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2424&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2581&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2700&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2815&$top=100
Got page containing 100 values with continuation token .../queues?api-version=2021-11-01&$skip=2915&$top=100
Got page containing 85 values with continuation token 
Count with waiting: 1985

I wouldn't even call this a throttling issue per-se and I don't think it's something the SDK can fix. It seems to be a bug in the API endpoint. @HarveyLink, do you agree, and if so, do you know how we could bring this to the attention of the relevant team?

If this is the case, I agreed this should be a service issue. I suggest you could open an Azure support request. It should get you in touch with the the Azure Service Team.

HarveyLink avatar Mar 21 '24 02:03 HarveyLink

OK. Thanks for your assessment. We have an open support case about not being able to submit support requests 🙃, but will submit this issue once it's fixed.

kimsey0 avatar Mar 21 '24 14:03 kimsey0

My support case about not being able to submit support requests for Azure Service Bus resources has been solved, and I have submitted this issue for investigation. As mentioned above, I believe this to be a problem with the API, not with the SDK, but let's see if the relevant team has any findings before closing this issue.

kimsey0 avatar Apr 03 '24 14:04 kimsey0