dagster icon indicating copy to clipboard operation
dagster copied to clipboard

[rfc] Dedupe definitions by reference equality.

Open dpeng817 opened this issue 8 months ago • 10 comments

There are cases where users can easily end up with the same assets definition object floating around in multiple places. Consider the following case:

  • asset my_asset is in module module_1
  • asset check my_check on my_asset is in module_2. This module imports my_asset from module 1.
  • A user calls load_... on module_1 and module_2. In their top level Definitions object, we will throw an error for duplicate asset keys.

The typical fix pattern here is actually pretty annoying. You either need to manually import all assets, or completely reorganize your project structure. Instead, we should just perform the deduping for them if there's reference equality across assetsdefinitions objects. We actually already do this in the individual load_... functions.

Added a new test for the deduping behavior and assume I'll need to fix a few test breakages that this will cause.

dpeng817 avatar Jun 20 '24 23:06 dpeng817