dlt
dlt copied to clipboard
Feat/1331 disables deduplication for incremental
Description
This PR disables deduplication for the test case described by ingestr here: https://github.com/dlt-hub/dlt/issues/971#issuecomment-1983417044
Related Issues
- Resolves #1131
Questions
@rudolfix
- I could not understand your points 2, 3, and 4 in the issue #1131 . Are they already implemented in this PR?
- I am not sure if the last test
tests/extract/test_incremental.py::test_deduplication_on_write_disposition_not_merge
makes sense at all because onwrite_disposition="replace"
the table is truncated before the load. I included it because the ticket speaks of the merge write disposition so I wanted to test the opposite too. Feel free to drop this commit. - Are the assertions on the incremental's
last_value
superfluous? - I could not find a way to test that the incremental returns all values without deduplication. Thus, I implemented the tests by making assertions on the loaded data. Is this strategy fine?
TODO after merge
- [ ] notify the developers of ingestr