mongo-connector
mongo-connector copied to clipboard
Implement #728 Flush doc managers before oplog progress write
This is my proposal for issue #728. Basically it gives us the ability to use an async or buffering doc manager (such as kafka) and ensure that writes are sent prior to writing oplog progress.
In our Kafka Doc Manager, the implementation essentially looks like:
def flush(self):
self.producer.flush()
It is a different purpose than commit()
, because with an async or buffering doc manager data can be lost without the flush()
We also implemented something like this in our own branch. It's critical that the oplog progress is only updated after all docmanagers have committed. I'm not sure why it already isn't. It caused us significant data loss.