ServiceBusExplorer
ServiceBusExplorer copied to clipboard
paolosalvatori
Purge DLQ is not clearing DLQ when there are many messages
We are trying to use this tool to clean up some DLQ's that have hundreds of thousands of messages and the tool is not handling it as I would expect it to. I am using the latest version downloaded from here, v.5.0.18.
It is taking a very long time and is only purging a small number of messages from the queue. At this rate it would take many many hours to purge the DLQ. This is a Premium service bus. Server and Receive timeout are set to 60s.
@ththiem, what's the average message size? Are you using
I suspect this is due to how the functionality is implemented to work around several issues with ASB.
To "purge" a queue, SBE has to receive the messages in ReceiveAndDelete mode. It has to account for session vs non-session messages, track the progress, etc. And if you have hundreds of thousands of messages, I can see that it might not be very speedy.
The good news is that the native purge is being worked on and is planned for the first half of 2024 (https://github.com/Azure/azure-service-bus/issues/1). Once the feature is released, SBE will be updated to take advantage of it.
The size is not that bad. Session-based entities - that might be a bit of a problem. Looking at the code, it's only one session at a time that's being purged https://github.com/paolosalvatori/ServiceBusExplorer/blob/f356ffbd2a88f1f0cf03bb9815657a66f080bfa2/src/ServiceBus/Helpers/ServiceBusPurger.cs#L99 And prefetch is set to 10 messages only https://github.com/paolosalvatori/ServiceBusExplorer/blob/f356ffbd2a88f1f0cf03bb9815657a66f080bfa2/src/ServiceBus/Helpers/QueueServiceBusPurger.cs#L44-L54 This would explain why it's not very performant.
I'd say it's worth waiting for the native purge feature. Unless you can't wait and would like to send a PR.
I agree that adding more sessions, if possible, would speed up the purging. However, I doubt that increasing the prefetch count will have any positive effect.
@ththiem, does it stop purging even though the queue is not empty?
Edit: Added a question, removed unnecessary advice
I tested purging > one million messages on a Premium queue without issues.
Could it be that your computer went into sleep mode?
If not, I think there is something wrong with your queue. Try changing a non-critical property such as "Enable Batched Operations" and Update. Then change it back and Update again. That should recreate the backend and with luck you'll get a better one.
If that does not work either, I recommend you to create a case with Microsoft.