connector-x icon indicating copy to clipboard operation
connector-x copied to clipboard

Error when trying to read values from Postgresql for array type column

Open PalakTiwary1394 opened this issue 2 years ago • 1 comments

What language are you using?

Python.

What version are you using?

3.9

What database are you using?

PostgreSQL

What dataframe are you using?

Pandas

Can you describe your bug?

I am trying to read values from Postgresql using connectorx into a pandas dataframe but it is throwing the following error.

df=cx.read_sql(connect_string, q) File "C:\Users\palak\anaconda3\envs\brmv\lib\site-packages\connectorx_init_.py", line 224, in read_sql result = _read_sql( pyo3_runtime.PanicException: not implemented: _text

What are the steps to reproduce the behavior?

Read a column that holds a text array using connectorx

Database setup if the error only happens on specific data or data type

This is happening as the columns named 'buckets' and 'sizes' hold a text array. image (1)

Example query / code
    df = pd.DataFrame()
    connect_string = "postgres://otjoiayz:[email protected]/otjoiayz"
    q = "SELECT id, title, gender, product_type, buckets, sizes FROM products"
    df=cx.read_sql(connect_string, q)

What is the error?

image (3)

PalakTiwary1394 avatar Jan 11 '23 02:01 PalakTiwary1394

Encountered the same problem. See https://github.com/sfu-db/connector-x/blob/f1d4e5f985f107f4c1cd52a10f668e9ee0be182e/connectorx/src/transports/postgres_arrow.rs#L36-L66

current implementation of pg to arrow data types conversion does not seem to support nested types at all. So types like int[] float[] varchar[] will also cause panics.

And here's the arrow2 implementation, the Vec types are present. https://github.com/sfu-db/connector-x/blob/f1d4e5f985f107f4c1cd52a10f668e9ee0be182e/connectorx/src/transports/postgres_arrow2.rs#L36-L74

Could try with arrow2 backend.

Cabbagec avatar Jun 06 '23 06:06 Cabbagec