dagster icon indicating copy to clipboard operation
dagster copied to clipboard

Add `executed_with_root_target` condition to handle partial runs / failures

Open OwenKephart opened this issue 4 months ago • 2 comments

Summary & Motivation

Resolves: https://github.com/dagster-io/dagster/issues/24389

To put it simply, this ensures the any_deps_updated condition does not "count" an update from a parent if the run that materialized that parent also planned to materialize the child.

This guards against a variety of situations where a run can be launched with the intention of updating many assets, but fail to do so for whatever reason (intentional skip, failure event). In those cases, the assumption is that the child should NOT be attempted again (even though the parent did successfully materialize).

This is a fairly niche implementation detail that users should not have to think about when crafting their own policies.

How I Tested These Changes

I added a parameterized test that failed before this change (for the skip and fail cases) and passes now.

Changelog

Fixed an issue with AutomationCondition.eager() that could cause it to attempt to launch a second attempt of an asset in cases where it was skipped or failed during a run where one of its parents successfully materialized.

OwenKephart avatar Oct 14 '24 17:10 OwenKephart