ibis
ibis copied to clipboard
feat(api): add rows construct to support SQL `VALUES` statements
This PR adds support for a SQL VALUES-like construct, called Rows in ibis.
The API entry point is ibis.rows which returns a Table expression.
The motivation is to allow use of simple in memory datasets in expressions.
ibis.rows handles type inference which makes it suitable for handling the
case of inserting into/creating tables from Python objects.
Notes
API Stuff
- Iterables of list/tuples/dicts are supported.
ibis.rowshas both acolumnsargument and aschemaargument. IfschemaisNone, the function performs type inference on the entire dataset.- Things become unusably slow somewhere between 10k and 200k rows depending on the number of columns. This seems fine to me.
- When both
columnsandschemaareNoneand the row type is not a mapping the default naming scheme iscol{i}where i is the zero-indexed ith column number. This template can be changed using an optionrows_default_column_name_template.
Backend Specific Stuff
- ClickHouse doesn't seem to support the notion of
VALUES. It might be posible to implement it byarrayJoin-ing an array ofTupletypes. I left that as a follow up. - MySQL and SQLite don't support
SELECT-ing from aVALUESstatement directly, because SQLAlchemy compiles these incorrectly to a form that these two dialects of SQL don't support. Expressions based onibis.rowswon't work those backends.con.create_tableworks and is tested for both. - I didn't implement impala support for this, and I won't until someone who isn't a maintainer actively asks for it :)
- PySpark translation is implemented.
- Pandas and Dask are not implemented, these are in theory straightforward.
cc @tswast
Closes #3832.
Test Results
6 files 6 suites 3m 50s :stopwatch: 3 184 tests 3 170 :heavy_check_mark: 14 :zzz: 0 :x: 9 552 runs 9 510 :heavy_check_mark: 42 :zzz: 0 :x:
Results for commit 94b17f70.
:recycle: This comment has been updated with latest results.
Hello @cpcloud! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:
Comment last updated at 2022-08-25 10:35:31 UTC
Codecov Report
Merging #4323 (5e186ed) into master (9796d4a) will decrease coverage by
0.10%. The diff coverage is88.58%.
:exclamation: Current head 5e186ed differs from pull request most recent head 94b17f7. Consider uploading reports for the commit 94b17f7 to get more accurate results
@@ Coverage Diff @@
## master #4323 +/- ##
==========================================
- Coverage 92.68% 92.58% -0.11%
==========================================
Files 181 181
Lines 20555 20694 +139
Branches 2928 2950 +22
==========================================
+ Hits 19052 19159 +107
- Misses 1136 1166 +30
- Partials 367 369 +2
| Impacted Files | Coverage Δ | |
|---|---|---|
| ibis/backends/pyspark/__init__.py | 86.00% <ø> (ø) |
|
| ibis/expr/schema.py | 80.55% <21.73%> (-11.18%) |
:arrow_down: |
| ibis/backends/base/sql/compiler/select_builder.py | 97.86% <75.00%> (-0.34%) |
:arrow_down: |
| ibis/expr/format.py | 93.10% <75.00%> (-0.71%) |
:arrow_down: |
| ibis/expr/rules.py | 88.21% <75.60%> (-1.93%) |
:arrow_down: |
| ibis/expr/api.py | 97.34% <93.33%> (-0.31%) |
:arrow_down: |
| ibis/backends/base/sql/alchemy/__init__.py | 90.27% <100.00%> (+0.38%) |
:arrow_up: |
| ibis/backends/base/sql/alchemy/query_builder.py | 93.44% <100.00%> (+0.10%) |
:arrow_up: |
| ibis/backends/base/sql/compiler/query_builder.py | 96.72% <100.00%> (+0.03%) |
:arrow_up: |
| ibis/backends/duckdb/compiler.py | 97.36% <100.00%> (+0.07%) |
:arrow_up: |
| ... and 7 more |