[watermill-amqp] Deadlock on Delivery Acknowledgement Timeout

[prochac]

Hi, we run into issue that Watermill AMQP subscriber stucks. We use managed RabbitMQ by AWS. It was recently upgraded and introduced this timeout. https://forums.aws.amazon.com/thread.jspa?messageID=994285 https://www.rabbitmq.com/consumers.html#acknowledgement-timeout

All what I know for now, is that when the ack times out then watermill blocks on channel's lock. Here: https://github.com/streadway/amqp/blob/75d898a42a940fbc854dfd1a4199eabdc00cf024/channel.go#L1548

Here is a example what reproduces the issue https://github.com/prochac/ThreeDotsLabs_watermill_I242

[AlexCuse]

Not sure the best way to handle the channel exceptions in watermill context (may be able to use a separate channel to pump errors through) but there is an AckWaitTimeout used in the final select within processMessage in the nats client that I think could be helpful here.