dagster icon indicating copy to clipboard operation
dagster copied to clipboard

Split freshness implementation, descriptions

Open dpeng817 opened this issue 10 months ago • 1 comments

The shared freshness implementation felt very hard to parse. Even though there was some shared code, it was largely a series of if-else statements, and bugs were slipping through due to obfuscation from this.

Instead, outside of shared utility functions, switch to having separate implementations for the two freshness builder methods.

Open to potentially consolidating in the future, but felt like we were currently in the worst of both worlds.

Along the way, I also switched up the descriptions and metadata to more accurately affect the freshness definition being addressed. The new metadata fields are as follows:

  • dagster/last_cron_tick_timestamp: if using cron-based freshness, the last completed tick of the cron.
  • dagster/lower_bound_timestamp: the earliest that we expect the latest record to have arrived in order for the asset to be considered fresh.
  • got rid of dagster/expected_by; we felt it was confusing to think of the failure case in terms of a deadline. I could be convinced to make the time partition freshness descriptions a little less verbose.

There are a few associated test changes as a result of changes to description, errors, and metadata.

dpeng817 avatar Apr 19 '24 23:04 dpeng817

  • #21320 Graphite 👈
  • #21295 Graphite
  • #21251 Graphite
  • #21247 Graphite
  • master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @dpeng817 and the rest of your teammates on Graphite Graphite

dpeng817 avatar Apr 19 '24 23:04 dpeng817