Ravi Gawai

Closing this issue since there is new similar [Issue](https://github.com/databrickslabs/dlt-meta/issues/94) which is implemented using dlt **direct publishing mode**

dlt-meta takes approach of looking at target tables to check your flows , this is closely aligned to dlt implementation. You can fork and override merge function so that you...

You can use `extract_select_exp` function to generate select_exp array for silver_transformations.json file generation then feed to dlt-meta onboarding job which will generated bronze dataflowspec. Check below example ``` from pyspark.sql.types...

@pavelilyushko , You need to do it once before running onboarding, generate your `silver_transformations.json `using above function then load `dataflowspecs` using onboarding-job. Since you know the schema before in hand,...

We can add bring your own transformations functionality so that you can add columns

@Lackshu @WilliamMize Would this be done now with last release supporting below features: [Added support for file metadata columns for autoloader](https://github.com/databrickslabs/dlt-meta/pull/56) [Bring your own custom transformations for bronze/silver layer](https://github.com/databrickslabs/dlt-meta/issues/68)

@sathya-reddy-m we are adding databricks labs cli integration first, also I added databricks SDK way deploying onboarding job and dlt pipeline. In coming future will add support for DAB.

Implementation Details: **Onboarding:** 1. Introduce **snapshot** format inside onboarding file 2. Introduce **bronze_apply_changes_from_snapshot** config **keys** and **scd_type** are mandatory fields ``` "bronze_apply_changes_from_snapshot":{ "keys": ["id"] "scd_type": "1" "track_history_column_list": [] "track_history_except_column_list":[] }...

@mweirath see if [this](https://github.com/databrickslabs/dlt-meta/tree/issue_90) fix resolves it! If you are using autoloader you dont need to provide schema ddl and apply_changes would work for evolving schema.

@mweirath you can close it with comments! thanks!