Hyukjin Kwon

icon indicating a website link https://g.co/kgs/mtcWskC icon indicating an email [email protected]

icon company @Apache @Databricks icon location South Korea icon location Comments from this profile are opinions of the profile owner. ASF member, Apache Spark PMC member and committer @apache Staff Software Engineer @databricks

Results 330 comments of Hyukjin Kwon

Groupby apply with as_index=False

It's because of missing type hint support for index. This is a limitation for now. @kismsu you can simply omit the type hints and let it infers properly for now.

why will a single head trigger a window function?

Weird. Window should not be used in this case. can you share your full codes?

why will a single head trigger a window function?

@feiwofeifeixiaowo, Just to confirm, does it still show a window even when you set `ks.set_option('compute.default_index_type', 'distributed')`? It would be easier to analyze further if you have a self-contained reproducer.

Implement Pandas rolling window with options such as center and win_type

Yeah, we should implement them.

pandas fixed width file support

Koalas is now in Apache Spark officially. Let's file an issue there. From a cursory look, looks like we can implement it by 1. distributing input StringIO, 2. reading any...

pandas fixed width file support

cc @xinrong-databricks and @itholic since you guys are triaging the issues.

Write custom metadata to output files with dataframe.to_parquet?

Can we file a JIRA in Apache Spark JIRA (https://issues.apache.org/jira/projects/SPARK)? This repository is in maintenance mode

Implement 'weights' and 'axis' in sample at DataFrame and Series

@itholic can you review this please?

Distributed index in pandas on pyspark does not work as expected

Yeah, you described all correctly. The order isn't guaranteed in general. Can you try to turn on `compute.ordered_head` configuration?

Implement to_sql

Yup, you can also leverage [DataFrame.to_spark_io](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_spark_io.html) and [koalas.read_spark_io](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.read_spark_io.html#databricks-koalas-read-spark-io) to read and write from/to external DBMSs. You might want to take a look at [databricks.koalas.sql](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.sql.html) too.