dlt
dlt copied to clipboard
add lag / attribution window to incremental
Background In many cases certain portion of data should be reacquired during incremental loading. Ie we want to always capture last 7 days of data when getting daily analytics report. Or we want to refresh slack message replies and we have a moving window of 7 days for that. Technically we would always pass start_date +/- the lag to the function accepting incremental
Requirements
-
- [ ] add new optional filed to
Incremental
class that will hold the lag.
- [ ] add new optional filed to
-
- [ ] lag should be expressed as float and will be interpreted depending on the type of the cursor. for datetimes it is a lag value in seconds, for any other type use + / - operator depending on the
last_value_func
- [ ] lag should be expressed as float and will be interpreted depending on the type of the cursor. for datetimes it is a lag value in seconds, for any other type use + / - operator depending on the
-
- [ ] we support only
min
andmax
forlast_value_func
. for customs we do not have "+" operator defined.
- [ ] we support only