[QUERY] Azure.Messaging.ServiceBus performance issues ?

[MarienMonnier]

Library name and version

Azure.Messaging.ServiceBus 7.11.1

Query/Question

I'm trying to migrate from Microsoft.Azure.ServiceBus v5.0.0 to Azure.Messaging.ServiceBus v7.11.1, but I'm seeing huge performance drops in my use-case. I'm not sure if I'm doing something wrong or not.

I have a service bus instance with ~1500 queues ; ~700 topics, and I need to query once all queues/topics/subscriptions to do some monitoring on a subset of it later.

My previous code using ManagementClient looks like this, where I first get all queues, then all topics/subscriptions.

private ManagementClient _managementClient = new ManagementClient("Endpoint=sb://REDACTED.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REDACTED");

private async Task<List<string>> DiscoverQueuesAsync(string[] prefixes)
{
    const int count = 100;

    var result = new List<string>();
    var skip = 0;
    IList<QueueDescription> callReturn;
    do
    {
        callReturn = await _managementClient.GetQueuesAsync(count, skip);
        result.AddRange(callReturn
            .Where(qd =>prefixes.Any(p => qd.Path.StartsWith($"{p}/", StringComparison.InvariantCultureIgnoreCase)))
            .Select(qd => qd.Path));
        skip += count;
    } while (callReturn.Any());

    return result;
}

private async Task<TopicsAndSubscriptions> DiscoverTopicsAndSubscriptions(string[] topicPrefixes, string[] queuePrefixes)
{
    const int count = 100;

    var result = new TopicsAndSubscriptions
    {
        Subscriptions = new List<Subscription>(),
        Topics = new List<string>()
    };
    var skip = 0;
    IList<TopicDescription> currentTopics;
    do
    {
        currentTopics = await _managementClient.GetTopicsAsync(count, skip).ConfigureAwait(false);
        foreach (var currentTopic in currentTopics))
        {
            if (topicPrefixes.Any(prefix => currentTopic.Path.StartsWith(prefix, StringComparison.OrdinalIgnoreCase)))
            {
                result.Topics.Add(currentTopic.Path);
            }

            var subscriptions = await _managementClient.GetSubscriptionsAsync(currentTopic.Path).ConfigureAwait(false);

            result.Subscriptions.AddRange(subscriptions
                .Where(s => s.ForwardTo != null && queuePrefixes.Any(prefix => new Uri(s.ForwardTo).AbsolutePath.StartsWith($"/{prefix}/", StringComparison.OrdinalIgnoreCase)))
                .Select(s => new Subscription(s.TopicPath, s.SubscriptionName, new Uri(s.ForwardTo).AbsolutePath.Substring(1))));
        }

        skip += count;
    } while (currentTopics.Any());

    return result;
}

After moving to the new Azure.Messaging.ServiceBus package, my code looks like this, so I'm trying to use new Azure Identity (DefaultAzureCredential) with my local account (just in case this changes something).

private ServiceBusAdministrationClient _administrationClient = new ServiceBusAdministrationClient("REDACTED.servicebus.windows.net", new DefaultAzureCredential(new DefaultAzureCredentialOptions
        {
            ExcludeManagedIdentityCredential = true
        }));

private async Task<List<string>> DiscoverQueuesAsync(string[] prefixes)
{
    var result = new List<string>();

    var pageableQueues = _administrationClient.GetQueuesAsync();
    await foreach (var queue in pageableQueues)
    {
        if (prefixes.Any(p => queue.Name.StartsWith($"{p}/", StringComparison.InvariantCultureIgnoreCase)))
        {
            result.Add(queue.Name);
        }
    }

    return result;
}

private async Task<TopicsAndSubscriptions> DiscoverTopicsAndSubscriptions(string[] topicPrefixes, string[] queuePrefixes)
{
    var result = new TopicsAndSubscriptions
    {
        Subscriptions = new List<Subscription>(),
        Topics = new List<string>()
    };

    var pageableTopics = _administrationClient.GetTopicsAsync();
    await foreach (var topic in pageableTopics)
    {
        if (topicPrefixes.Any(prefix => topic.Name.StartsWith(prefix, StringComparison.OrdinalIgnoreCase)))
        {
            result.Topics.Add(topic.Name);
        }

        var subscriptions = _administrationClient.GetSubscriptionsAsync(topic.Name);
        await foreach (var subscription in subscriptions)
        {
            if (subscription.ForwardTo != null && 
                queuePrefixes.Any(prefix => new Uri(subscription.ForwardTo).AbsolutePath.StartsWith($"/{prefix}/", StringComparison.OrdinalIgnoreCase)))
            {
                result.Subscriptions.Add(new Subscription(subscription.TopicName, subscription.SubscriptionName, new Uri(subscription.ForwardTo).AbsolutePath.Substring(1)));
            }
        }
    }

    return result;
}

Regarding performances when I launch my application:

• With Microsoft.Azure.ServiceBus
  • DiscoverQueues: 4 618ms
  • DiscoverTopicsAndSubscriptions: 18 628ms
• With Azure.Messaging.ServiceBus:
  • DiscoverQueues: 7 792ms
  • DiscoverTopicsAndSubscriptions: 127 779ms

Instead of starting in 23s, my apps needs more than 2 minutes with the new SDK! I tried several times, on different days, but I always see the same performance drop when using the new SDK.

I tried to use Benchmark.net to compare performances of the two. But when I do, the 2 SDK have the same performances (about 5s for DiscoverQueues and 17s for DiscoverTopicsAndSubscriptions), it seems like the fact that Benchmark.net does warmup and stuff, the issue is not seen.

I then tried to execute the DiscoverQueues + DiscoverTopicsAndSubscriptions methods two times in a row (while creating a new ServiceBusAdministrationClient between the two), but in the same application run: 1st DiscoverQueues: 11 374ms 1st DiscoverTopicsAndSubscriptions: 122 431ms 2nd DiscoverQueues: 7 058ms 2nd DiscoverTopicsAndSubscriptions: 17 235ms

It looks like the issue is only present for the first calls, is something cached after?

I'm not sure if this is related to the SDK, or if it is related to Service Bus when using an Azure Identity?

Let me know if I can provide more things

Environment

No response

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist. Please note that responses will be delayed due to the US holidays.

[JoshLove-msft]

The fact that the lag only occurs on the first calls is suggestive of this being a caching issue involving the credentials. Can you try enabling SDK logging so we can see what is happening under the hood in the Azure.Identity library?

Another thing to try to further isolate this as the issue would be to use connection string auth and see if the issue persists.

[MarienMonnier]

Hello Josh,

Thanks for getting back to me.

I enabled SDK logging and I can see that (for example with the Subscriptions endpoint):

• with an UMI the first call to all Subscriptions endpoints takes between 0.2 and 0.5s
• with an UMI, the second call to any Subscriptions endpoint that was already called takes less than 0.1s
• with a connection string using a shared access key, all calls takes less than 0.1s

Here are the logs: ServiceBusPerf_AccessKey.txt ServiceBusPerf_UMI.txt I redacted the service bus & subscriptions names

As I have about 900 calls to /Subscriptions, this takes a large amount of time.

It seems like on Service Bus, they cache the credentials for each distinct call? So this issue may not be with SDK, but maybe more with Service Bus itself. If this is the case, do you know where I should contact them? Thanks

[JoshLove-msft]

I think you are probably right about the potential bottleneck being on the service. The only times that the client is interacting with the token service (based on the logs) is prior to the initial request for each client that you created. That means any extra time would have to be due to how the service is dealing with the authorization. I'm reaching out to the service team to find out if this is expected.

[JoshLove-msft]

I received confirmation that this is expected - connection string auth is all within Service Bus so it is expected that it will be faster for the first request for a resource as compared to using managed identity. With managed identity, Service Bus needs to talk to the token service. After the initial request, it caches the token, hence the lower latency for subsequent requests for the same resources.

[ghost]

Hi @MarienMonnier. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.