Hyukjin Kwon

Sure, that sounds making sense.

Someone tried in https://github.com/databricks/koalas/pull/2020. I think we should fix this as now PyArrow also supports Python 3.9.

I will take a look soon

Yeah.

Just my guts but I think it's because `ARROW_PRE_0_15_IPC_FORMAT` is not set (see also https://github.com/databricks/koalas#getting-started)

Thanks, @lopez-

Yes, the issue https://issues.apache.org/jira/browse/ARROW-7986 can be closed. #1324 fixed the original issue. ```python >>> import pandas as pd >>> import databricks.koalas as ks >>> from pyspark.ml.linalg import SparseVector >>> >>>...

@prakharsandhu mind reporting the issue in Apache Spark https://issues.apache.org/jira/projects/SPARK/issues? this repo is in maintenance mode

> spark isn't able to handle rolling operations across partitions either unless you redefine the partitions yourself (again, please correct me if I am wrong.) Yes, what you said is...

Seems like a bug ..