spark-snowflake icon indicating copy to clipboard operation
spark-snowflake copied to clipboard

spark connector for snowflake unable to handle NULLs in numeric columns

Open prabodh1194 opened this issue 1 year ago • 1 comments

I have a glue job using the snowflake spark connector. It keeps failing when it sees null values in the dataframe. I am unable to debug this issue.

Glue 4 is using this snowflake driver -- spark-snowflake_2.12-2.12.0-spark_3.3.jar

error:
 Numeric value 'null' is not recognized

prabodh1194 avatar Feb 19 '24 06:02 prabodh1194

Hello @prabodh1194 ,

Thanks for raising the issue.

I tried to reproduce the issue, but its working fine. A table with numeric column having null. created a dafaframe out of it. ` df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("dbtable", "my_table_int").load()

#Show DataFrame content df.show()`

Ouptut:

+----+-----+ | ID| NAME| +----+-----+ | 1| John| | 2|Alice| |null| null| | 4| Bob| |null| null| +----+-----+ Could you provide the code snippet ( small sample application) so we can reproduce the issue.

Regards, Sujan

sfc-gh-sghosh avatar Apr 01 '24 04:04 sfc-gh-sghosh