kamu-cli
kamu-cli copied to clipboard
sparkmagic garbles some dataset types
sparkmagic currently transforms SQL queries into spark code with resulting dataframe transferred in stringified form. Since type information is lost pandas tries to guess the datatypes and often does it incorrectly. We need to update sparkmagic to include type information in the transmitted results and use it when building pandas dataframe.
Need to check if latest sparkmagic fixes this