Wenchen Fan comments

Results 245 comments of


                                            Wenchen Fan

[SPARK-54442][SQL][FOLLOWUP] Add codegen for TIME numeric conversion functions

thanks, merging to master!

[SPARK-53446][CORE] Optimize BlockManager remove operations with cached block mappings

thanks, merging to master!

[SPARK-33863][PYTHON] Respect session timezone in udf workers

can we add a test which sets `spark.sql.session.timeZone` to a different value than the CI machine local timezone?

[SPARK-33863][PYTHON] Respect session timezone in udf workers

> unless we are able to read session config inside the conversion function Where do we do the conversion? At lease for the UDF case the conversion should all happen...

[SPARK-33863][PYTHON] Respect session timezone in udf workers

@gaogaotiantian The key of Spark `TimestampType` is that it's an absolute timestamp. The session timezone only matters when we render the timestamp without timezone (e.g. `df.show`, or cast to string,...

[SPARK-33863][PYTHON] Respect session timezone in udf workers

ah this is tough. I agree with "always map TimestampType with aware datetime", but it can be a breaking change to python UDFs, as it's not only a data change,...

[SPARK-33863][PYTHON] Respect session timezone in udf workers

Let's clearly define this "strict mode". For the input side, Spark can be more lenient and allow mixed timezone aware and non-aware datetime objects, by using session timezone. For the...

[SPARK-33863][PYTHON] Respect session timezone in udf workers

ok so Spark should fail if naive and aware datetimes are mixed? It's another breaking change though...

[SPARK-33863][PYTHON] Respect session timezone in udf workers

Let’s open a PR so we can discuss the details more closely.

[SPARK-54556][CORE] Rollback succeeding shuffle map stages when shuffle checksum mismatch detected

thanks, merging to master/4.1!