py-questdb-client icon indicating copy to clipboard operation
py-questdb-client copied to clipboard

Add ability for user to set `create_table` and `table_partition` when saving data to QDB

Open newskooler opened this issue 8 months ago • 0 comments

Hi,

I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.

I use the below code to save new and old data. If the table exists - it appends to it. If it does not, it creates it and appends to it.

import pandas as pd
from questdb.ingress import Sender

df = pd.DataFrame({
    'id': pd.Categorical(['toronto1', 'paris3']),
    'temperature': [20.0, 21.0],
    'humidity': [0.5, 0.6],
    'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.dataframe(df, table_name='sensors', at='timestamp')

This causes a few issue:

  1. upon creation it always creates a partition by DAY

Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency

  1. It silently creates a table by default, even if one does not exist. Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.

This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.

newskooler avatar Jun 12 '24 11:06 newskooler