databrickslabs
Chain bronze and silver layer in same pipeline
Provide option while launching dataflowpipeline by which both bronze/silver can be launched under single dlt.
Once Direct publishing mode becomes PuPr then can be merged to release branch.
Use CLI to test this feature. We have added broze_silver layer option which will launch single dlt with bronze/silver layers.
Introducing new layer option as bronze_silver for invoking dataflowpipeline invoke_dlt_pipeline function.
This feature will only work if customer create DLT pipelines using Direct publishing mode feature which is currently in private preview.
While creating DLT-META pipeline please select Direct publishing mode then provide layer=bronze_silver
Launch DLT-META pipeline using CLI: Integrated this feature in CLI which will have option if customer wants to launch bronze_silver dlt pipeline.
Step1: databricks labs dlt-meta onboard Step2: databricks labs dlt-meta deploy
This will launch bronze and silver layer into single DLT pipeline as below: