duckdb_engine icon indicating copy to clipboard operation
duckdb_engine copied to clipboard

[Bug]: Long column names are truncated

Open cpcloud opened this issue 1 year ago • 3 comments

What happened?

Observed in https://github.com/ibis-project/ibis/issues/8393.

Long column names appear to be truncated when using duckdb_engine to fetch results.

Reproducible example:

import sqlalchemy as sa

long_name = "a_very_very_very_very_very_very_very_very_very_long_column_name"

eng = sa.create_engine("duckdb:///test.db")

with eng.begin() as con:
    con.exec_driver_sql(f"CREATE OR REPLACE TABLE t ({long_name} INT)")

t = sa.Table("t", sa.MetaData(), autoload_with=eng)
t1 = sa.select(t.c[0]).select_from(t).subquery()
query = sa.select(t1.c[long_name])

with eng.begin() as con:
    result = con.execute(query)
    batch = result.connection.connection.fetch_record_batch()

assert batch.schema.names[0] == long_name, batch.schema.names[0]

DuckDB Engine Version

0.10.0

DuckDB Version

0.9.2, 0.10.0

SQLAlchemy Version

1.4.51

Relevant log output

No response

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

cpcloud avatar Feb 20 '24 14:02 cpcloud

Looks like this is a default inherited from Postgres: https://github.com/sqlalchemy/sqlalchemy/blob/1c58fe53b6fd069cbb82955ddaf9eb5405076146/lib/sqlalchemy/dialects/postgresql/base.py#L2941

Mause avatar Feb 20 '24 14:02 Mause

I wonder if this is another nail in the coffin for subclassing from that dialect.

In the next Ibis release we're moving away from SQLAlchemy entirely to avoid the extra layer and myriad dependencies (which Ibis already has plenty of!).

duckdb_engine has been great to work with!

cpcloud avatar Feb 20 '24 14:02 cpcloud

Given that it's in the "base" Postgres dialect, rather than the psycopg2 one, I'm not sure it is

I saw in the linked issue, congrats!

Mause avatar Feb 20 '24 14:02 Mause