woodwork
woodwork copied to clipboard
Remove `backup_dtype` from categorical logical types and bump pyspark version if necessary
With version 1.8.0, Koalas added support for the category
dtype. We should bump our min Koalas version to 1.8.0 and update all of our categorical LogicalTypes to remove the backup_dtype
attribute.
I think this will also allow us to remove the use of backup_dtype
completely from the Woodwork code, including the usage in the LogicalType
class.
This will also require updating several tests.
This may also allow us to remove LogicalType._get_valid_dtype
and just use LogicalType.primary_dtype
(though now that there's no backup dtype, maybe we should change the name of "primary_dtype")
@tamargrey sounds good. We will be addressing this issue in the upcoming sprint.
@rwedge ice-boxing this for now.