azure-service-bus
azure-service-bus copied to clipboard
Azure
Feature: peek messages based on status
Allow to query messages based on their state rather than receiving messages in all states and filtering on the client side.
From the comment this could be an option for the Premium Tier as that one uses customers dedicated MU pools.
@SeanFeldman we put it on our backlog and consider it for our planning.
@SeanFeldman This is in the backlog, but not a planned item. There are higher priority items that need more immediate attention. From the looks of it, it might take a while before we get to this feature.
Sean commented on an SO question of mine where this would have helped (https://stackoverflow.com/questions/53710350/azure-service-bus-can-i-view-scheduled-messages/53713041#53713041)
Essentially we're using service bus as a lightweight task queue, with tasks able to be scheduled to run at a future time and a function app running as tasks become available. I'd like to be able to build a UI around this to show upcoming tasks without independently maintaining a list of all the tasks and when they're scheduled, and this would probably be the most straightforward way to do that.
Another use case to have support for browsing based on some sort of criteria (not entirely based on message status, but similar need in a criteria to be passed to the broker). Raised by @cz-kmocherla
It will be great if we can have this feature. This feature will be very useful from support prospective.
Wouldn't receiving them using the sequence number have the same effect as a purge.
This feature will help improve scaling logic used by Function Apps - https://github.com/Azure/Azure-Functions/issues/715
Any ETA for the fix? cc @chiangvincent
We have a situation where there are like 3 million messages in a deadletter queue dating years back, because we don't always maintain them, knowing we can eventually just purge it.
This is a huge problem when an 'incident' occurs, and we need to replay a few recent deadletters. Assuming we want to replay them in a non-destructive way (leaving them in the DL queue, so we can repeat the operation if something goes wrong mid-way), we have to peek through the entire DL queue from the beginning just to reach more recent records at the end. And with partitioned queues, I assume it reads one partition at a time, so 'newer' messages are actually at the end of each partition.
The point is, we have to read past a lot of old stuff many times to get to the new stuff when peeking non-destructively through the queue. Built-in filtering capabilities (i.e. by enqueue date at the very least) would be immensely useful.
With standard tier's 256KB batch limit, It took us over 21 hours to read through messages. Of course, the VPN cut out at 21 hours when it was only 90% through inspecting messages, so now we have to retry the whole operation. We didn't keep a list of which sequence numbers were visited, but probably will in the future, so we at least know what we already replayed if it fails partway through. Also, with the message browsing order apparently being non-deterministic (I've not confirmed this, but I've read that it starts with "whichever message broker happens to respond first" for partitioned queues), we can't just skip or read past the number of messages we've already read. No idea how many messages are in each broker, and so on. It's a real pain.
With custom server-side filtering, at least we'd be able to locate the messages we're interested in (i.e. only ones with enqueue times during the incident time window.).
First, I must mention, we could use multiple readers if we were actually removing messages as we read them, because concurrent readers with locks won't be served the same messages. But we're not using lock or delete modes. We're peeking non-destructively; we want to leave the message there in case something goes wrong part way through the replay operation, as it did.
So, assuming peek-only mode, the reason we have to read with only a single thread is that if we were to use multiple QueueClients, that would result in each one re-reading all the same messages; no way to filter messages or coordinate amounts them, at least not out of the box. A workaround may be to craft starting sequence numbers with explicit partition bytes, such that we can run 16 QueueClients in parallel, each targeting a different partition, and stopping each one when it reaches the end of the partition. I'm assuming we can detect that when we receive messages with a different partition number than what we started with. With a clever sequence-number-range leap-frogging algorithm, we could probably even have more than 16 readers by having N readers blasting through a partition without stepping on each other's ranges. Of course, I'm not sure if this even applies to Premium namespaces which apparently don't support partitioning? Documentation at that deep of a level is lacking, but it's really necessary to efficiently peek through a queue.
Alternatively, we might just have to do a pass where we read the message normally, and sort them into two separate queues (one for old messages we're not interested in and one for new ones). But it would be nice to just be able to browse the queue in-place. Also, it would be nice to selectively delete messages in a queue. There's currently no way to do that. Cannot defer messages without acquiring them through normal peek/lock mode (so no way to read-past messages you don't want to defer), and we cannot 'receive' (by sequence number) messages that haven't been deferred. So we can't just browse, get a sequence number, then 'receive it'. I has to have been located through normal queue order. It's all extremely limiting what we can do with queued messages.
This has surfaced to the top of our backlog, so expect more information in the coming months, although we don't have specific timelines yet.
Thank you for your feedback on this item. We are currently doing active development on this feature, and expect to have more to share around its release in the next couple of months.
how are we doing on this? i have 8 scheduled messages that i have no idea when they will deliver and have no way to cancel them or modify their schedule.
some day i hope to know what they are.
