Add support for dlt.apply_changes_from_snapshot

[ravi-databricks]

Provide support for dlt.apply_changes_from_snapshot

[ravi-databricks]

Implementation Details: Onboarding:

1. Introduce snapshot format inside onboarding file
2. Introduce bronze_apply_changes_from_snapshot config keys and scd_type are mandatory fields

"bronze_apply_changes_from_snapshot":{
      "keys": ["id"] 
      "scd_type": "1"
      "track_history_column_list": []
      "track_history_except_column_list":[]
}

DataflowPipeline:

1. Add argument to dataflowpipeline to accept snapshot_reader_func
2. snapshot_reader_func will be applied to dlt.apply_changes_from_snapshot while doing bronze write

Usage:

1. Provide snapshot reader function in a notebook while invoking Dataflowpipeline:
2. Introduce new method

pip install dlt-meta

import dlt
from src.dataflow_spec import BronzeDataflowSpec

def exist(path):
    try:
        if dbutils.fs.ls(path) is None:
            return False
        else:
            return True
    except:
        return False


def next_snapshot_and_version(latest_snapshot_version, dataflow_spec):
    latest_snapshot_version = latest_snapshot_version or 0
    next_version = latest_snapshot_version + 1    
    bronze_dataflow_spec: BronzeDataflowSpec = dataflow_spec
    options = bronze_dataflow_spec.readerConfigOptions
    snapshot_format =  bronze_dataflow_spec.sourceDetails["snapshot_format"]
    snapshot_root_path = bronze_dataflow_spec.sourceDetails['path']    
    snapshot_path = f"{snapshot_root_path}{next_version}.csv"
    if (exist(snapshot_path)):
        snapshot = spark.read.format(snapshot_format).options(**options).load(snapshot_path)
        return (snapshot, next_version)
    else:
        # No snapshot available
        return None 

layer = spark.conf.get("layer", None)
from src.dataflow_pipeline import DataflowPipeline
DataflowPipeline.invoke_dlt_pipeline(spark, layer, next_snapshot_and_version=next_snapshot_and_version)