dlt icon indicating copy to clipboard operation
dlt copied to clipboard

add lag / attribution window to incremental

Open rudolfix opened this issue 1 year ago • 0 comments

Background In many cases certain portion of data should be reacquired during incremental loading. Ie we want to always capture last 7 days of data when getting daily analytics report. Or we want to refresh slack message replies and we have a moving window of 7 days for that. Technically we would always pass start_date +/- the lag to the function accepting incremental

Requirements

    • [ ] add new optional filed to Incremental class that will hold the lag.
    • [ ] lag should be expressed as float and will be interpreted depending on the type of the cursor. for datetimes it is a lag value in seconds, for any other type use + / - operator depending on the last_value_func
    • [ ] we support only min and max for last_value_func. for customs we do not have "+" operator defined.

rudolfix avatar Feb 14 '24 17:02 rudolfix