dagster
dagster copied to clipboard
dagster-io
[DS][27/n] Make candidate_slice and true_slice Optional
Summary & Motivation
This helps avoid a class of errors using the type system. We have these concepts of "AssetSubsets" vs. "ValidAssetSubsets", where valid asset subsets are known to have been created with the asset's current-tick PartitionsDefinition, and AssetSubsets have gone through a serdes layer and therefore may have a differen partitions definition than they did at serialization time.
This means that care needs to be taken when using a bare AssetSubset.
Previously, we had these fancy operators which would treat AssetSubsets which had invalid partitions_definitions as the empty subset, but that is not actually always accurate. i.e. you may have been keeping track of all the materialized partitions of an asset using a subset, then you update the partitions definition. The correct thing to do here is to recompute the materialized partitions from scratch, NOT assume that this susbet is equivalent to the empty subset.
By explicitly creating a None value for candidate_slice / true_slice in these cases, we force authors of conditions to handle these cases through the type system. This already uncovered a potential bug in the new UpdatedSinceCron condition, in which it would return an empty set in the case that a partition was updated since the latest cron tick, but then the partitions definition was updated.
How I Tested These Changes
-
#21641
: 2 dependent PRs (#21670 , #21671 ) -
#21640
-
#21648 👈
-
#21615
-
#21613
-
#21612
-
#21573
-
#21546
-
#21545
-
#21541
-
#21540
-
#21539
-
#21538
-
#21537
-
#21536
-
#21535
-
#21521
-
#21520
-
#21511 : 1 other dependent PR (#21512 )
-
#21510
-
#21508
-
#21507
-
#21505
-
#21504
-
#21503
-
#21502
-
#21501
-
#21500
-
#21499
-
master
This stack of pull requests is managed by Graphite. Learn more about stacking.
Join @OwenKephart and the rest of your teammates on Graphite