datasets icon indicating copy to clipboard operation
datasets copied to clipboard

Support pyarrow LargeListType

Open Modexus opened this issue 1 year ago • 2 comments

Fixes #6834

Modexus avatar Apr 24 '24 11:04 Modexus

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixed the conversion from pyarrow to python Sequence features.

There is still an issue that if features are passed the Sequence always forces conversion to ListArray. This probably causes issues if the LargeListArray is actually needed.

There doesn't seem to be a great solution since this list is created solely on the schema for Sequence. One solution would be to always use LargeListArray instead.

Modexus avatar Apr 24 '24 13:04 Modexus

I am retaking this PR because we would like to have this feature implemented.

albertvillanova avatar Jul 01 '24 08:07 albertvillanova