dagster
dagster copied to clipboard
Split freshness implementation, descriptions
The shared freshness implementation felt very hard to parse. Even though there was some shared code, it was largely a series of if-else statements, and bugs were slipping through due to obfuscation from this.
Instead, outside of shared utility functions, switch to having separate implementations for the two freshness builder methods.
Open to potentially consolidating in the future, but felt like we were currently in the worst of both worlds.
Along the way, I also switched up the descriptions and metadata to more accurately affect the freshness definition being addressed. The new metadata fields are as follows:
-
dagster/last_cron_tick_timestamp
: if using cron-based freshness, the last completed tick of the cron. -
dagster/lower_bound_timestamp
: the earliest that we expect the latest record to have arrived in order for the asset to be considered fresh. - got rid of
dagster/expected_by
; we felt it was confusing to think of the failure case in terms of a deadline. I could be convinced to make the time partition freshness descriptions a little less verbose.
There are a few associated test changes as a result of changes to description, errors, and metadata.
This stack of pull requests is managed by Graphite. Learn more about stacking.
Join @dpeng817 and the rest of your teammates on Graphite