mongo-kafka icon indicating copy to clipboard operation
mongo-kafka copied to clipboard

Allow Source Connector to send tombstones on delete events when publishing only fullDocument

Open cosimomeli opened this issue 3 years ago • 2 comments

Hi, we had the need to send tombstones on delete events to allow consumers to delete data when origin data in deleted. Currently the Source connector filters out every null fullDocument. I added a configuration tombstone.on.delete to opt-in the feature. When both tombstone.on.delete and publish.full.document.only are true, the delete event with null fullDocument will be sent as tombstone (null value and null schema) to Kafka. An integrationTest has been created to check for null values.

cosimomeli avatar May 02 '22 10:05 cosimomeli

Tracked at https://jira.mongodb.org/browse/KAFKA-312

cosimomeli avatar May 10 '22 15:05 cosimomeli

This feature is essential for maintaining a compacted Kafka topic that mirrors the contents of a MongoDB collection using publish.full.document.only. In addition to space savings, tombstones are also important for data privacy reasons if the documents contain private data.

denisw avatar Jun 02 '22 09:06 denisw

Had to manually rebase and merged See: 759966ec4eca869bf640678b3be6bd1899bdc07f

Note: I renamed the configuration option to: publish.full.document.only.tombstone.on.delete to be super explicit and to follow the convention in relating the configuration to the higher level config (publish.full.document.only).

Many thanks @cosimomeli for the PR and all those that commented on the ticket.

rozza avatar Dec 15 '22 10:12 rozza