Hyukjin Kwon

icon indicating a website link https://www.linkedin.com/in/hyukjin-kwon-25045412b icon indicating an email [email protected]

icon company @Apache @Databricks icon location South Korea icon location Comments from this profile are opinions of the profile owner. ASF member, Apache Spark PMC member and committer @apache Staff Software Engineer @databricks

Results 207 comments of Hyukjin Kwon

[SPARK-38098][PYTHON] Add support for ArrayType of nested StructType to arrow-based conversion

@LucaCanali thanks for addressing my comment. The test seems like failing: ``` ====================================================================== ERROR [0.182s]: test_pandas_array_struct (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/__w/spark/spark/python/pyspark/sql/tests/test_pandas_udf_scalar.py", line 152, in test_pandas_array_struct result...

[SPARK-38292][PYTHON]Support na_filter for pyspark.pandas.read_csv

@itholic can you review this one please?

[SPARK-38292][PYTHON]Support na_filter for pyspark.pandas.read_csv

Hey, I think the fix here is too hacky. Can we make this working independently with other options being set?

[SPARK-39822][PYTHON][PS] Provide a good feedback to users

cc @zhengruifeng @xinrong-meng @itholic FYI

[SPARK-33782][K8S][CORE]Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

cc @dongjoon-hyun @tgravescs @Ngone51 FYI

[SPARK-40019][SQL] Refactor comment and logic of ArrayType's containsNull

Yeah, I am fine with the current approach.

[WIP][SPARK-38493][PS][FOLLOWUP] Improve more coverage for Pandas API on Spark

Merged to master.

[SPARK-39976][SQL] ArrayIntersect should handle null in left expression correctly

cc @rednaxelafx FYI

[SPARK-39755][CORE] Randomization in Spark local directory for K8 resource managers

Yeah, doesn't seem related. You can retrigger the failed tests for doubly sure rough

[SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter

cc @viirya and @HeartSaVioR mind taking a look please when you find some time?