deletion during incremental syncs

[simon-weber]

Feature Description

Tools like Hightouch have the option to delete rows in the destination during an incremental sync after they're deleted from the source. As far as I can tell, that's not something Sling supports right now. You either need to run a full refresh / truncate, or do an incremental sync + some kind of post-run cleanup.

[flarco]

Yea, deletes are a bit tricky. The way Hightouch does this currently is actually inefficient because it does a full select of the source data each time, whereas sling adds a {incremental_where_cond} clause in the query to filter at the source system. Hightouch is able to detect missing IDs by caching a list of the values in S3 and comparing them after doing a full select. But since Sling only selects new data, it doesn't have a complete list of the IDs to do a whole comparison.

I'm thinking something that could work for Sling would be to only SELECT the primary key values in the target table AND the source table, and then do a comparison to see which ones need to be deleted in the target table...

[flarco]

This is added. Please see https://docs.slingdata.io/examples/database-to-database/incremental#delete-missing-records-soft-hard.

Closing.