connector-x
connector-x copied to clipboard
partition_range is not taken into account in memory allocation
When you partition a query and specify partition_range it is taken into account to partition the query but not in the memory allocation.
Reproduce:
connectorx.read_sql(
'postgres://postgres:postgres@localhost:5432/postgres',
'select * from lineitem',
partition_num=4,
partition_on='l_orderkey',
partition_range=(1, 10),
return_type='pandas',
)
If we check the logs:
...
[2025-05-29T17:08:26Z DEBUG tokio_postgres::prepare] preparing query s1: SELECT count(*) FROM (SELECT * FROM lineitem) AS CXTMPTAB_COUNT
...
[2025-05-29T17:08:26Z DEBUG connectorx::pandas::dispatcher] Allocate destination memory: 1199969x16
...
It will call a count(*) and allocate memory for the entire dataset.