Redelivered Flag Not Exposed

[samartha-kar23]

trafficstars

Feature request

Expose Redelivered to consumer listening for messages on channel.

Problem: Redelivered Flag Not Exposed

According to the RabbitMQ reliability docs, messages may be redelivered in the event of network or node failures. RabbitMQ sets the redelivered flag to indicate that a message may have been previously delivered. If this flag is not set, the message is guaranteed to be new.

Currently, Watermill’s AMQP subscriber does not expose this redelivered flag in the message metadata, which means that consumers using Watermill cannot reliably detect if they are processing a retried (redelivered) message.

Real-World Impact

Not exposing the redelivered flag has tangible implications for production systems:

• Inefficient Duplicate Handling: Without access to the redelivered flag, consumers must implement deduplication based solely on message IDs or other less efficient means. This results in additional processing overhead and increases the complexity of the consumer logic.
• Risk of Message Duplication: In scenarios where a connection interruption leads to a message being redelivered (and its corresponding negative acknowledgment lost), the consumer might inadvertently reprocess messages that were already handled. This can lead to duplicate business operations, which is critical in systems that require exactly-once or idempotent processing.
• Complex Error Recovery: During network or node failure events, the ability to differentiate fresh versus retried messages is crucial for implementing effective error recovery strategies. If the redelivered flag is unavailable, it becomes challenging to properly manage message acknowledgments and rejections, potentially leading to mismanagement of message state and increased operational burden.
• Operational Overhead: As a workaround to the missing flag, developers may resort to building custom deduplication mechanisms, which adds complexity and increases the chance of bugs or performance issues in a production environment.

Example use case

Expose the redelivered flag in message.Metadata using a dedicated key, such as amqp_redelivered, so that consumers can:

• Efficiently detect redelivery cases.
• Implement optimized handling logic for duplicate messages.
• Simplify error recovery and acknowledgment procedures in challenging network conditions.

How it can look like in code

in https://github.com/ThreeDotsLabs/watermill-amqp/blob/d55465059d3d5d434881026a27d09b7b162641d7/pkg/amqp/subscriber.go#L385

// Redelivered flag is set to true when the message is redelivered.
msg.Metadata.Set("amqp_redelivered", strconv.FormatBool(amqpMsg.Redelivered))
if amqpMsg.Redelivered {
	s.logger.Trace("Message redelivered", msgLogFields)
}

Consumer code

  for {
      select {
      case message := <-messages:
...
          if message.Metadata.Get("amqp_redelivered") == "true" {
              log.Warn().Msgf("Message %s is redelivered, ignoring", messageID)
          }
...
      }
  }