dagster
dagster copied to clipboard
[DS][49/n] Update static constructors to use Since operator
Summary & Motivation
As title -- now we can take advantage of the "since" operator in a variety of different ways to simplify and improve our on_cron
and eager
"composite conditions".
On the on_cron
side, note that the previous definition had a bad property, which is that if the requested materialization failed, a new one would be immediately re-requested (as nothing would have changed in the eyes of the system -- the parents would still be newer than the cron tick, the child would still be older than the cron tick, and there'd no long be an in-progress run). eager
had handling for this case (temporarily removed downstack), but it was ugly (basically just a fixed retry interval, which is conceptually confusing).
This shows off the value of the new since
operator -- you can just make sure that you've requested a materialization of an asset since the newest time a condition became true, which means that you can guarantee that the cron condition will only become true once per cron tick, because the CronTickPassed condition only becomes true once per cron tick. Similarly, the any_dep_matches(newly_updated) condition will only be true at the instant a parent is updated, so you can guarantee a maximum of one downstream run per parent update, regardless of if that downstream run fails etc.
This basically reimplements the current AMP eager behavior, but in a much more principled way, as we assign actual operators and names to these bookkeeping operations, rather than relying on some rather arcane codepaths to magically do a similar thing.