spark-expectations icon indicating copy to clipboard operation
spark-expectations copied to clipboard

Published 20 hours ago • verified publisher icon Nike-Inc

[BUG] Fails on Databricks runtime 14 and above

Open diviteja-g opened this issue 5 months ago • 0 comments

Describe the bug Spark expectations fails on databricks runtime 14 and above with following error

SparkExpectationsMiscException: Input rules_df is not of dataframe type

To Reproduce Steps to reproduce the behavior: Run any spark expectation job on Databricks runtime 14 and above

Expected behavior Spark expectations job doesn't fail on Databricks runtime 14 and above

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: iOS
  • Browser: chrome
  • Version

Additional context

Databricks returns Dataframe of type pyspark.sql.connect.dataframe.DataFrame instead of pyspark.sql.Dataframe in some cases which results in this error as current version of Spark Expectations strictly expects pyspark.sql.Datafram type as input.

All other operations that can be done on pyspark.sql.Datafram can also be done on pyspark.sql.connect.dataframe.DataFrame

diviteja-g avatar Aug 29 '24 21:08 diviteja-g