WilliamMize
WilliamMize
I did this in my own cloned branch. Inside of read_dlt_cloud_files: spark.readStream.format(bronze_dataflow_spec.sourceFormat) .options(**reader_config_options) .schema(schema) .load(source_path) .withColumn("_filePath",input_file_name()) .withColumn("_loadDate",lit(datetime.now())) I thought about making a pull request for this but didn't want it...
@ravi-databricks Thank you, I have implemented 0.0.8 and this now supports our use case.
We see the same thing in our lineage. Our bronze streaming table and silver streaming table are in their own schema(bronze and silver), not sure if this breaks it or...