spark-snowflake
spark-snowflake copied to clipboard
spark connector for snowflake unable to handle NULLs in numeric columns
I have a glue job using the snowflake spark connector. It keeps failing when it sees null values in the dataframe. I am unable to debug this issue.
Glue 4 is using this snowflake driver -- spark-snowflake_2.12-2.12.0-spark_3.3.jar
error:
Numeric value 'null' is not recognized
Hello @prabodh1194 ,
Thanks for raising the issue.
I tried to reproduce the issue, but its working fine. A table with numeric column having null. created a dafaframe out of it. ` df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("dbtable", "my_table_int").load()
#Show DataFrame content df.show()`
Ouptut:
+----+-----+ | ID| NAME| +----+-----+ | 1| John| | 2|Alice| |null| null| | 4| Bob| |null| null| +----+-----+
Could you provide the code snippet ( small sample application) so we can reproduce the issue.
Regards, Sujan