apache
Cancel MessageProcessor while allowing in-flight messages to succeed
Is your feature request related to a problem? Please describe
I am looking to leverage KEDA to support auto-scaling my Pulsar consumers based on topic volumes. One of the best practices with regards to properly handling requests to scale down would involve preventing the MessageProcessor from receiving new messages while allowing any messages that are currently in-flight (or have already been processed but not yet acknowledged) from continuing and sending the acknowledgement back to the broker. After reviewing the code, I can't seem to find a way to handle that.
Describe the solution you'd like and alternatives you've considered
Add support for a mechanism to prevent the MessageProcessor from picking up new messages while allowing already pulled messages to continue. In the meantime, I can leverage scale as job to work-around this issue.
Hi @smbecker Just to make sure I understand, you want a "nice shutdown" functionality sort of like Complete+Completion on ISendChannel?
Hi @smbecker Just to make sure I understand, you want a "nice shutdown" functionality sort of like Complete+Completion on ISendChannel?
I don't think it really has anything to do with ISendChannel. What I was looking for was a way to provide a different CancellationToken used for Process than the one used for processing each message or acknowledging processed messages. The idea being that when MessageProcessor.Process is cancelled, then any messages that have already been received (and processing started) could continue to process until completion and then be acknowledged before ultimately terminating.
Hi @smbecker. If you want you can compile DotPulsar and try the new ShutdownGracePeriod feature. https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG.md If you do, please tell us how it works for you.
No feedback received. 3.4.0 with the ShutdownGracePeriod feature will release soon.