Hyukjin Kwon

icon indicating a website link https://www.linkedin.com/in/hyukjin-kwon-25045412b icon indicating an email [email protected]

icon company @Apache @Databricks icon location South Korea icon location Comments from this profile are opinions of the profile owner. ASF member, Apache Spark PMC member and committer @apache Staff Software Engineer @databricks

Results 207 comments of Hyukjin Kwon

[SPARK-39941][PYTHON][PS] window and min_periods should be integer in rolling func

cc @zhengruifeng @xinrong-meng @itholic FYI

[SPARK-39833][SQL] Remove partition columns from data schema in the case of overlapping columns to fix Parquet DSv1 incorrect count issue

Yeah, let's just get this in first.

[SPARK-39833][SQL] Remove partition columns from data schema in the case of overlapping columns to fix Parquet DSv1 incorrect count issue

Merged to master, branch-3.3, and branch-3.2.

Predicate Pushdown not Working

@Lukas012 do you mind reporting a issue in https://issues.apache.org/jira/projects/SPARK?

Implement Koalas Missing APIs

Please go ahead @AishwaryaKalloli!

Minor fix to docs for read_csv

Thanks, @tnixon. This library has been merged into Apache Spark: https://github.com/apache/spark. Would you mind proposing the PR there?

Implement resample from pandas

Should be able to via using [`DataFrame.apply_batch`](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.apply_batch.html) for now.

pandas.merge_asof

I think this can be implemented when cogroup feature is implemented in Spark side, which is in progress.

pandas.merge_asof

I roughly suspect is right after Spark 3.0 is released. Won't be too long since Spark 3.0 preview was released as well.

pandas.merge_asof

I think we should implement this in Spark 3.2, cc @itholic and @xinrong-databricks FYI