mongo-connector icon indicating copy to clipboard operation
mongo-connector copied to clipboard

Implement #728 Flush doc managers before oplog progress write

Open travisp opened this issue 7 years ago • 1 comments

This is my proposal for issue #728. Basically it gives us the ability to use an async or buffering doc manager (such as kafka) and ensure that writes are sent prior to writing oplog progress.

In our Kafka Doc Manager, the implementation essentially looks like:

def flush(self):
  self.producer.flush()

It is a different purpose than commit(), because with an async or buffering doc manager data can be lost without the flush()

travisp avatar Aug 16 '17 18:08 travisp

We also implemented something like this in our own branch. It's critical that the oplog progress is only updated after all docmanagers have committed. I'm not sure why it already isn't. It caused us significant data loss.

makhdumi avatar Oct 13 '17 16:10 makhdumi