woodwork icon indicating copy to clipboard operation
woodwork copied to clipboard

Remove `backup_dtype` from categorical logical types and bump pyspark version if necessary

Open thehomebrewnerd opened this issue 3 years ago • 3 comments

With version 1.8.0, Koalas added support for the category dtype. We should bump our min Koalas version to 1.8.0 and update all of our categorical LogicalTypes to remove the backup_dtype attribute.

I think this will also allow us to remove the use of backup_dtype completely from the Woodwork code, including the usage in the LogicalType class.

This will also require updating several tests.

thehomebrewnerd avatar Aug 10 '21 14:08 thehomebrewnerd

This may also allow us to remove LogicalType._get_valid_dtype and just use LogicalType.primary_dtype (though now that there's no backup dtype, maybe we should change the name of "primary_dtype")

tamargrey avatar Aug 31 '21 15:08 tamargrey

@tamargrey sounds good. We will be addressing this issue in the upcoming sprint.

gsheni avatar Aug 31 '21 15:08 gsheni

@rwedge ice-boxing this for now.

gsheni avatar Sep 14 '21 13:09 gsheni