py-questdb-client
py-questdb-client copied to clipboard
Add ability for user to set `create_table` and `table_partition` when saving data to QDB
Hi,
I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data. If the table exists - it appends to it. If it does not, it creates it and appends to it.
import pandas as pd
from questdb.ingress import Sender
df = pd.DataFrame({
'id': pd.Categorical(['toronto1', 'paris3']),
'temperature': [20.0, 21.0],
'humidity': [0.5, 0.6],
'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
sender.dataframe(df, table_name='sensors', at='timestamp')
This causes a few issue:
- upon creation it always creates a partition by
DAY
Feature request: add a parameter which is by default DAY
but can be changed by the user to be MONTH
, NONE
or other partition frequency
- It silently creates a table by default, even if one does not exist.
Feature request: add a parameter which is by default
True
(to create a table) but can be also set toFalse
such that if a table does not exist, it will not create one.
This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.