dagster
dagster copied to clipboard
RFC: Class-based dlt integration
Summary & Motivation
This is a follow up to our discussion yesterday in product staff to show how this would look in practical application.
The idea here is a new API for definition factories. Instead of decorator-based approaches.
This PR also includes a new Specs
class whose purpose is to group specs together and apply operations across them.
A few advantages here, which I will speak to inline comments as well.
- The default case is strictly instantiating a single class, rather than having to copy and paste a decorated function with a generator etc. This is simpler.
- There is no need for a translator. Instead you just create specs and pass them around. I have included the
Specs
helper class to assist with applications of properties across many specs. - When you need to write custom code typing is much clearer as you are overriding a function rather than than implicit typing contracts that our decorators impose.
- Separation of metadata and compute. Clarifies things both large and small. It's small thing but the fact that computations and specs each have separate
tags
attributes rather thamop_tags
onmulti_asset
communicates a much more clear mental model.
How I Tested These Changes
BK
Changelog [New | Bug | Docs]
NOCHANGELOG