dagster icon indicating copy to clipboard operation
dagster copied to clipboard

RFC: Class-based dlt integration

Open schrockn opened this issue 6 months ago • 2 comments

Summary & Motivation

This is a follow up to our discussion yesterday in product staff to show how this would look in practical application.

The idea here is a new API for definition factories. Instead of decorator-based approaches.

This PR also includes a new Specs class whose purpose is to group specs together and apply operations across them.

A few advantages here, which I will speak to inline comments as well.

  • The default case is strictly instantiating a single class, rather than having to copy and paste a decorated function with a generator etc. This is simpler.
  • There is no need for a translator. Instead you just create specs and pass them around. I have included the Specs helper class to assist with applications of properties across many specs.
  • When you need to write custom code typing is much clearer as you are overriding a function rather than than implicit typing contracts that our decorators impose.
  • Separation of metadata and compute. Clarifies things both large and small. It's small thing but the fact that computations and specs each have separate tags attributes rather tham op_tags on multi_asset communicates a much more clear mental model.

How I Tested These Changes

BK

Changelog [New | Bug | Docs]

NOCHANGELOG

schrockn avatar Aug 22 '24 17:08 schrockn