wren-engine icon indicating copy to clipboard operation
wren-engine copied to clipboard

Fix: MSSQL add N char

Open ansutung opened this issue 3 months ago • 3 comments

Summary by CodeRabbit

  • New Features

    • Added connectors for Redshift, Postgres, SQL Server, and Canner.
    • DuckDB can attach additional databases and list/attach files from S3 or local storage.
    • Query results now return as Apache Arrow tables for improved performance and compatibility.
    • Automatic handling of Decimal and UUID types for consistent outputs.
  • Bug Fixes

    • Unified, clearer error messages across connectors; safer dry-run behavior.
    • More robust connection closing to prevent crashes or double-closing.
    • MSSQL correctly handles Unicode string literals.
    • Preserved timeout semantics while improving invalid SQL detection.

ansutung avatar Oct 03 '25 04:10 ansutung

Walkthrough

Adds multiple connectors (Redshift, Postgres, MSSQL, Canner), changes query return type to PyArrow tables, introduces unified error handling, implements close semantics, enhances DuckDB behaviors, and adds utilities for UUID/Decimal post-processing and PostgreSQL type-name retrieval in ibis-server/app/model/connector.py.

Changes

Cohort / File(s) Summary of Changes
Connector additions
ibis-server/app/model/connector.py
Introduces RedshiftConnector, PostgresConnector, MSSqlConnector, and CannerConnector with query/dry_run implementations and close methods.
API return type updates
ibis-server/app/model/connector.py
Updates Connector.query and SimpleConnector.query to return pa.Table instead of pandas DataFrame.
Unified error handling
ibis-server/app/model/connector.py
Wraps backend exceptions into WrenError with ErrorCode/ErrorPhase; adjusts mappings (e.g., timeouts vs INVALID_SQL) across query and dry_run.
PyArrow post-processing
ibis-server/app/model/connector.py
Adds _handle_pyarrow_unsupported_type, _cast_uuid_columns, _round_decimal_columns in SimpleConnector to cast UUIDs to string and round Decimals to a fixed scale.
Close semantics and safety
ibis-server/app/model/connector.py
Adds close() methods to multiple connectors with safeguards to avoid double-closing and segfaults.
DuckDB enhancements
ibis-server/app/model/connector.py
Enables attaching additional DuckDB databases, listing/attaching files via S3/Local FS interfaces, and improves dry_run/close safety.
PostgreSQL type helper
ibis-server/app/model/connector.py
Adds _get_pg_type_names(connection) to retrieve PostgreSQL OID-to-type-name mappings.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Client
  participant Connector as SimpleConnector
  participant Ibis as Ibis Backend
  participant Arrow as PyArrow
  Note over Connector,Ibis: Query execution with PyArrow result
  Client->>Connector: query(sql, limit)
  Connector->>Ibis: compile/execute(sql, limit)
  Ibis-->>Connector: result (backend table)
  rect rgb(245,245,255)
    Note right of Connector: Post-process unsupported types
    Connector->>Connector: _handle_pyarrow_unsupported_type()
    Connector->>Connector: _cast_uuid_columns()
    Connector->>Connector: _round_decimal_columns(scale=9)
  end
  Connector->>Arrow: to_arrow()
  Arrow-->>Connector: pa.Table
  Connector-->>Client: pa.Table
sequenceDiagram
  autonumber
  actor Client
  participant Conn as AnyConnector
  participant Backend as Driver/DB
  participant Err as WrenError
  Note over Conn,Backend: Unified error handling for query/dry_run
  Client->>Conn: dry_run(sql) / query(sql)
  Conn->>Backend: execute / explain
  alt Success
    Backend-->>Conn: OK / result
    Conn-->>Client: Response
  else Backend raises (Trino/ClickHouse/psycopg/...)
    Backend-->>Conn: Exception
    rect rgb(255,245,245)
      Note right of Conn: Map to ErrorCode + ErrorPhase
      Conn->>Err: wrap(exception)
    end
    Err-->>Client: WrenError
  end
sequenceDiagram
  autonumber
  actor Client
  participant RS as RedshiftConnector
  participant Psy as psycopg Cursor
  participant Arrow as PyArrow
  Note over RS,Psy: Redshift autocommit + Python-level cursor
  Client->>RS: query(sql, limit)
  RS->>Psy: execute(sql)
  Psy-->>RS: rows / cursor data
  RS->>Arrow: to_arrow()
  Arrow-->>RS: pa.Table
  RS-->>Client: pa.Table
  Client->>RS: close()
  RS-->>Client: closed

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • Canner/wren-engine#1247 — Both touch DuckDBConnector: attach/list helpers and query return/signature changes align with this PR.
  • Canner/wren-engine#1207 — Shares _get_pg_type_names helper introduction and cursor/close handling in the same module.

Suggested labels

ibis, python

Suggested reviewers

  • douenergy
  • goldmedal

Poem

(\/) ( ••)✧ hop! New bridges built to data streams, UUIDs tamed, and Decimal gleams. Errors corralled, one burrowed voice— Arrow feathers our swift choice. Close the tunnel—quiet, poised.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 17.50% which is insufficient. The required threshold is 80.00%. You can run `@coderabbitai generate docstrings` to improve docstring coverage.
âś… Passed checks (2 passed)
Check name Status Explanation
Description Check âś… Passed Check skipped - CodeRabbit's high-level summary is enabled.
Title Check ✅ Passed The title mentions adding an N prefix for MSSQL Unicode literals, which corresponds to a valid subset of the changes. However, it does not reflect the primary objective of this pull request, which is the broad expansion of multi-backend support—including new Redshift, Postgres, MSSQL, and Canner connectors—along with enhanced PyArrow handling, unified error wrapping, and robust closing logic.
✨ Finishing touches
  • [ ] 📝 Generate Docstrings
đź§Ş Generate unit tests
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Oct 03 '25 04:10 coderabbitai[bot]

@ansutung, could you rebase on the latest main and add the description for what you want to propose?

goldmedal avatar Oct 03 '25 09:10 goldmedal

It may be a duplicate of https://github.com/Canner/wren-engine/pull/1338

goldmedal avatar Oct 03 '25 09:10 goldmedal