redflag icon indicating copy to clipboard operation
redflag copied to clipboard

Check for similar strings in categorical variables

Open kwinkunks opened this issue 3 years ago • 1 comments

Check edit distance to other unique values in low-cardinality categorical variables. (If high cardinality, might expect close labels, or might take too long to check all comparisons.)

E.g. to catch things like Sandstone / sandstone or shale / shales

Could also check for apparent abbreviations like fine sand / fs or limestone / limest.

kwinkunks avatar Sep 28 '22 17:09 kwinkunks

Potentially useful library here https://github.com/life4/textdistance

kwinkunks avatar Sep 28 '23 09:09 kwinkunks