dbt-duckdb icon indicating copy to clipboard operation
dbt-duckdb copied to clipboard

[Feature] Return the rows affected in the adapter response

Open rchui opened this issue 11 months ago • 3 comments

Other DBT adapters return in their response a field containing the "rows affected" and outputs a log like: [0 1 OK] to indicate the number of rows that were affected by an incremental / table model. This information is useful because these metrics can be harvested to track pipeline throughput and monitor for performance regressions.

It would be useful if dbt-duckdb also returned this.

rchui avatar Jan 07 '25 22:01 rchui

Ah yeah I wanted this back in the day when I first started working on the adapter, but IIRC the DuckDB python API didn't support it (i.e., it doesn't actually return the counts of affected rows for the statement via the rowcount field on the cursor object, it's just always -1.

I just tried it out on 1.1.3 and confirmed that it's still the case, and I don't think it's something that is easily fixable in DuckDB itself, see: https://github.com/duckdb/duckdb/issues/7924

jwills avatar Jan 10 '25 23:01 jwills

That's too bad. I did notice that in the duckdb CLI there is a dot command (.changes on|off) that does show how many rows are affected. Is there a reason this value is not just piped back through the Python API?

rchui avatar Feb 05 '25 23:02 rchui

I don't know, I don't work on the DuckDB Python API.

jwills avatar Feb 06 '25 15:02 jwills