dagster
dagster copied to clipboard
dagster-io
[DS][9/n] Add lookback_timedelta to InLatestTimeWindow condition
Summary & Motivation
This diverges slightly from past conversations we've had about separating out this functionality into two separate conditions.
That said, I think this definition actually ends up making more sense overall.
The most annoying bit with the originally-proposed definition is that if you define a "RecentPartitionsCondition" to mean "all partitions within the time window (current_time - 3 days, current_time)" you end up with the annoying result that this time range actually only contains 2 full daily partitions, as you are inevitably going to call this function at some time other than "exactly midnight", meaning that you have (partital_day, full_day, full_day, part of today) as your range, leaving you with only 2 partitions. I think this would be a persistant annoyance for users that would simply want to express "the last 3 daily partitions" in a natural way.
I think this formulation ends up being more intuitive, and has the added bonus of consolidating two very-similar concepts into a single condition.
This definition also handles partitions deifnitions with an end date nicely, which is nice.
How I Tested These Changes
-
#21511
: 2 dependent PRs (#21512 , #21520 ) -
#21510
-
#21508 👈
-
#21507
-
#21505
-
#21504
-
#21503
-
#21502
-
#21501
-
#21500
-
#21499
-
master
This stack of pull requests is managed by Graphite. Learn more about stacking.
Join @OwenKephart and the rest of your teammates on Graphite