connector-x icon indicating copy to clipboard operation
connector-x copied to clipboard

partition_range is not taken into account in memory allocation

Open surister opened this issue 5 months ago • 6 comments

When you partition a query and specify partition_range it is taken into account to partition the query but not in the memory allocation.

Reproduce:

connectorx.read_sql(
    'postgres://postgres:postgres@localhost:5432/postgres',
    'select * from lineitem',
    partition_num=4,
    partition_on='l_orderkey',
    partition_range=(1, 10),
    return_type='pandas',
)

If we check the logs:

...
[2025-05-29T17:08:26Z DEBUG tokio_postgres::prepare] preparing query s1: SELECT count(*) FROM (SELECT * FROM lineitem) AS CXTMPTAB_COUNT
...
[2025-05-29T17:08:26Z DEBUG connectorx::pandas::dispatcher] Allocate destination memory: 1199969x16
...

It will call a count(*) and allocate memory for the entire dataset.

surister avatar May 29 '25 17:05 surister