dlt icon indicating copy to clipboard operation
dlt copied to clipboard

Feat/1331 disables deduplication for incremental

Open willi-mueller opened this issue 4 months ago • 1 comments

Description

This PR disables deduplication for the test case described by ingestr here: https://github.com/dlt-hub/dlt/issues/971#issuecomment-1983417044

Related Issues

  • Resolves #1131

Questions

@rudolfix

  1. I could not understand your points 2, 3, and 4 in the issue #1131 . Are they already implemented in this PR?
  2. I am not sure if the last test tests/extract/test_incremental.py::test_deduplication_on_write_disposition_not_merge makes sense at all because on write_disposition="replace" the table is truncated before the load. I included it because the ticket speaks of the merge write disposition so I wanted to test the opposite too. Feel free to drop this commit.
  3. Are the assertions on the incremental's last_value superfluous?
  4. I could not find a way to test that the incremental returns all values without deduplication. Thus, I implemented the tests by making assertions on the loaded data. Is this strategy fine?

TODO after merge

  • [ ] notify the developers of ingestr

willi-mueller avatar Sep 28 '24 12:09 willi-mueller