azure-kusto-python icon indicating copy to clipboard operation
azure-kusto-python copied to clipboard

API CHANGE: add `to_dataframe` to each table

Open danield137 opened this issue 5 years ago • 2 comments

After investing work in https://github.com/Azure/azure-kusto-python/pull/124, and some internal discussions, we agreed to wait with this PR and reconsider changing the API to give better performance for both vanilla python and pandas use cases, and save some difficult trickery to allow parsing kusto type to dataframe:

Final api would look like

# result is of type KustoResultDataSet
result = client.execute(db, query)
# raw json 
result.tables[0].json()
# iterator with lazy parsing of json
result.tables[0].rows()
# dataframe parsing from raw json
result.tables[0].to_dataframe()

This will cause some memory pressure, so a best practice would probably be:

# either explicitly access a specific table and drop the reference after conversion
df = client.execute(db, query).primary_results[0].to_dataframe()
# or, parse it all
dfs = client.execute(db, query).to_dataframes()

Feel free to add your thoughts, code will be implemented in next couple of weeks.

danield137 avatar Apr 02 '19 10:04 danield137

https://github.com/Azure/azure-kusto-python/pull/127

danield137 avatar Apr 02 '19 12:04 danield137

Postponed

vladikbr avatar Aug 05 '20 14:08 vladikbr