mongo-kafka
mongo-kafka copied to clipboard
Allow Source Connector to send tombstones on delete events when publishing only fullDocument
Hi,
we had the need to send tombstones on delete events to allow consumers to delete data when origin data in deleted.
Currently the Source connector filters out every null fullDocument.
I added a configuration tombstone.on.delete to opt-in the feature.
When both tombstone.on.delete and publish.full.document.only are true, the delete event with null fullDocument will be sent as tombstone (null value and null schema) to Kafka.
An integrationTest has been created to check for null values.
Tracked at https://jira.mongodb.org/browse/KAFKA-312
This feature is essential for maintaining a compacted Kafka topic that mirrors the contents of a MongoDB collection using publish.full.document.only. In addition to space savings, tombstones are also important for data privacy reasons if the documents contain private data.
Had to manually rebase and merged See: 759966ec4eca869bf640678b3be6bd1899bdc07f
Note: I renamed the configuration option to: publish.full.document.only.tombstone.on.delete to be super explicit and to follow the convention in relating the configuration to the higher level config (publish.full.document.only).
Many thanks @cosimomeli for the PR and all those that commented on the ticket.