spark-expectations
spark-expectations copied to clipboard
[FEATURE] Enable Data Profiling using Spark Expectations
Is your feature request related to a problem? Please describe. Data Profiling feature helps to analyse the trends on the table overtime.
Describe the solution you'd like Explore profiling features in spark and implement via SE configuration
Describe alternatives you've considered NA
Additional context Databricks has Lakehouse Monitoring, which can be enabled via SE configuration on a table.