ibis icon indicating copy to clipboard operation
ibis copied to clipboard

feat: support `read_parquet`/`read_csv` for most backends

Open jcrist opened this issue 1 year ago • 3 comments
trafficstars

Currently we only support read_parquet for backends that have native support (like duckdb). In contrast, we support to_parquet for all backends, falling back to a common pyarrow implementation if a backend doesn't natively support it.

To provide more uniform feature coverage, we could write an equivalent common pyarrow (or other) implementation of our IO input methods (read_parquet/read_csv/...) that backends like postgres could fall back on.

jcrist avatar Jun 26 '24 21:06 jcrist

If duckdb is installed, there's also the option of using duckdb for faster support for parquet/csv table loading. I think that could be done in a 2nd pass, see #8110, but a pass using pyarrow first to cover all backends makes sense to me as a good first step.

jcrist avatar Jun 26 '24 21:06 jcrist

HI @jcrist I could take this one. Will do it one by one.

Plan to add read_parquet and read_csv. How about read_json and read_delta?

jitingxu1 avatar Jul 31 '24 16:07 jitingxu1

Any updates on this one?

csubhodeep avatar Sep 18 '24 09:09 csubhodeep