kamu-cli icon indicating copy to clipboard operation
kamu-cli copied to clipboard

sparkmagic garbles some dataset types

Open sergiimk opened this issue 6 years ago • 1 comments

sparkmagic currently transforms SQL queries into spark code with resulting dataframe transferred in stringified form. Since type information is lost pandas tries to guess the datatypes and often does it incorrectly. We need to update sparkmagic to include type information in the transmitted results and use it when building pandas dataframe.

sergiimk avatar Sep 12 '19 02:09 sergiimk

Need to check if latest sparkmagic fixes this

sergiimk avatar Aug 18 '21 02:08 sergiimk