azure-service-bus icon indicating copy to clipboard operation
azure-service-bus copied to clipboard

Azure Service Bus Topic metric shows incorrect total of Outgoing messages

Open MounicaChette opened this issue 2 years ago • 2 comments

Description

I am having a .net core webjob which will be reading messages from an Azure Topic in batches. I have encoutered a weird scenario while observing the metrics of Incoming VS Outgoing messages. Incoming messages was 10k and Outgoing messages were 64k. I have implemented logging to understand , if the webjob is processing the messages multiple times, but surprisingly the webjob was not processing messages more than once. FYI.. we I am using Microsoft.Azure.ServiceBus.Core.MessageReceiver to read messages from azure topic.

Actual Behavior

  1. Number of Incoming messages is not equal to number of outgoing messages.

Expected Behavior

  1. Number of Incomng messages is equal to number of outgoing messages.

Adding few screen shots of the issue: I have tested under a load of 10k messages. So our job is processing only 10k messages, but the metrics for outgoing messages is 64k

image

image

Code: image

MounicaChette avatar Aug 19 '21 06:08 MounicaChette

@shankarsama Can you look into this issue?

EldertGrootenboer avatar Apr 01 '22 18:04 EldertGrootenboer

We have assigned an engineer to look into this issue.

EldertGrootenboer avatar Aug 05 '22 20:08 EldertGrootenboer

We are suffering the same problem here, but the numbers are higher:

image

Using Azure Function trigger to process the subscriptions: Microsoft.Azure.WebJobs.Extensions.ServiceBus Version="3.2.0"

Is there any direction to understand this numbers?

viniciostorres avatar Nov 11 '22 12:11 viniciostorres

@MounicaChette https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide#flow-control You can find an explanation of the above scenarios (In metrics, the number of incoming messages is not equal to the number of outgoing messages) by clicking this link and selecting the transfer and flow control heading.

karankesri avatar Dec 20 '22 05:12 karankesri

Couple of reasons for this:

  1. Azure Functions app is configured with prefetch count or receive-batch and function app is slow in processing messages then the messages will be released/redelivered.
  2. Messages will be redelivered if messages are abandoned by the client or the message lock is lost due to service side updates.
  3. Client locked many messages but restarted before completing those messages.

shankarsama avatar Mar 17 '23 01:03 shankarsama