dlt icon indicating copy to clipboard operation
dlt copied to clipboard

Even more custom Destination follow up work

Open sh-rp opened this issue 1 year ago • 0 comments

Feature description

For properly releasing the generic destination we need to do the following:

  • [x] allow to set max_table_nesting in the decorator and pass it to normalizer via capabilities. set to 0 by default (no nesting)
  • [x] ignore dlt tables and remove dlt fields from data by default. (another setting in decorator)
  • [ ] make writing to files and other stateful operations easy. so I can open a file when table starts and close it when it ends. may be some kind of destination state with a flag available with current()
  • [ ] somehow support the destination lifecycle that regular destinations have (preferably without going OOO) so people can manage stateful objects, open and close files, truncate tables, even migrate schemas. we also need a "destructor" called when factory is garbage collected.
  • [ ] when callable on a factory is called, it should clone the factory (same behavior we have on resources)
  • [x] allow to control the number of load workers from device capabilities
  • [x] allow to sequence the load jobs from device capabilities (sequential, table - sequential, parallel)
  • [x] Create a nice example how to use it, e.g. with bigquery
  • [ ] Make the init command work, probably by just adding an empty decorated function as destination
  • [x] See if the loadpackage state tests are alright
  • [x] Decide on the name we actually want to use and promote. It probably will not be "generic destination"

regarding dlt init - we can use our init folder and add another variant that has a destination sink in it and use it when unknown destination (with unknown name) is requested. stuff will be trivial. we have AST parser and rewriter. my take is that @sultaniman will do it as a separate ticket

sh-rp avatar Mar 14 '24 08:03 sh-rp