Hyukjin Kwon

icon indicating a website link https://g.co/kgs/mtcWskC icon indicating an email [email protected]

icon company @Apache @Databricks icon location South Korea icon location Comments from this profile are opinions of the profile owner. ASF member, Apache Spark PMC member and committer @apache Staff Software Engineer @databricks

Results 330 comments of Hyukjin Kwon

GH-48442: [Python] Remove workaround that excluded struct types from `chunked_arrays`

@github-actions crossbow submit test-conda-python-3.11-hypothesis

GH-48442: [Python] Remove workaround that excluded struct types from `chunked_arrays`

Pushed again to retrigger the test. hyphothsis build itself passes (https://github.com/apache/arrow/pull/48443#issuecomment-3648388322)

GH-48442: [Python] Remove workaround that excluded struct types from `chunked_arrays`

Seems like: ``` tests/test_extension_type.py ................. [ 40%] Fatal Python error: Segmentation fault Current thread 0x0000000203059040 (most recent call first): File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/tests/test_fs.py", line 1224 in test_s3_options File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 166 in...

GH-48442: [Python] Remove workaround that excluded struct types from `chunked_arrays`

Thanks @adamreeve !

[SPARK-47008][CORE] Added Hadoops fileSystems hasPathCapability check to avoid FileNotFoundException(s) when using S3 Express One Zone Storage

can you fill the PR description please?

[MINOR][INFRA] Do not upload docker build record

cc @Yikun FYI

[SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used.

When the initial PR merged for json and csv mentioned in the PR (https://github.com/apache/spark/pull/33436), I realised that I underestimated the problem (https://github.com/apache/spark/pull/33436#discussion_r863271189). For reading actual files, there'd be much more...

[SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used.

We tried to fix this few times, e.g., https://github.com/apache/spark/pull/17293 and https://github.com/apache/spark/pull/14124. The breakage was pretty severe

[SPARK-50033] Add a hint to logical.Aggregate() node

Mind filing a JIRA please?

[SPARK-49985][SQL] Remove support for interval types in Variant

Merged to master.