Delivery Acknowledgement Timeout makes hutch process stuck
There is a timeout (by default 15 minute) for consumer to process and ack / nack a message. https://www.rabbitmq.com/consumers.html#acknowledgement-timeout
After this timeout if consumer still haven't done with the processing the consumer's channel would be closed, all the prefetched messages would be returned to the queue. However hutch consumer continues to work and if at this point the consumer finishes the message processing hutch will try to ack the message here:
https://github.com/ruby-amqp/hutch/blob/master/lib/hutch/worker.rb#L74
It will cause the Bunny::ChannelAlreadyClosed exception, because you can ack only via opened channel, however the error will be caught at the next line:
https://github.com/ruby-amqp/hutch/blob/master/lib/hutch/worker.rb#L75
and hutch will try to execute acknowledge_error(delivery_info, properties, @broker, ex) line. It will cause Bunny::ChannelAlreadyClosed again, because by default this line try nack the message, but again you can nack only on opened channel.
This new exception will be never reported by following line:
https://github.com/ruby-amqp/hutch/blob/master/lib/hutch/worker.rb#L77
and will be caught only in Bunny code here:
https://github.com/ruby-amqp/bunny/blob/c99ffcb0876a46d71c822b8b96a6ac978fd0af14/lib/bunny/channel.rb#L1769
printing a message to STDOUT, that means ignoring the error.
The main problem is the fact that hutch process will stay alive and will not receive new messages until the process restart.
is new about this issue or how to solve because I faced it when I use quorum queue when I do requeue!
Delivery acknowledgement timeout can be effectively disabled by bumping it to something like an hour or a few hours. It's not super safe but would work equally well for any tool.
Those who would like to see this addressed can see the detailed description of the exception flow above and contribute an improvement. This is open source software after all.