Message deduplication

Open hmarr opened this issue 12 years ago • 1 comments

When we're using mirrored queues, there is a chance of receiving a message multiple times. Prevent this by storing message ids / hashes in Redis or Zookeeper, and checking if messages have already been processed before handing them off to a consumer.

See https://github.com/xing/beetle/blob/master/lib/beetle/deduplication_store.rb

Apr 22 '13 12:04 hmarr

Oct 20 '13 16:10 ms-ati