datacompy icon indicating copy to clipboard operation
datacompy copied to clipboard

Published 20 hours ago • verified publisher icon capitalone

Datatype standardization before comparing for dataframes from DASK or Pyspark

Open xs005 opened this issue 11 months ago • 1 comments

In most cases, datacompy works very well for the comparison at the cell level. But I have met a few cases when I converted the datafrom from DASK or Pyspark dataframes, I need to apply the data type schema so that to get correct comparison result, especially the situation that you have long float64, and compare with another float32. The default data type from DASK or Pyspark dataframe may not correct. I would provide an example so that you can reproduce the issue.

xs005 avatar Mar 04 '24 17:03 xs005

I think a minimal example to show what issue you are facing would be very helpful for the team to understand the depth of the problem. So yes please.

fdosani avatar Mar 04 '24 17:03 fdosani

@xs005 Just wanted to follow up if you had a minimal example to understand the issue you are facing?

fdosani avatar Apr 08 '24 14:04 fdosani

Issue is stale. Closing. Please reopen if you have any updates.

fdosani avatar Apr 26 '24 17:04 fdosani