ibis icon indicating copy to clipboard operation
ibis copied to clipboard

feat(api): add rows construct to support SQL `VALUES` statements

Open cpcloud opened this issue 3 years ago • 2 comments

This PR adds support for a SQL VALUES-like construct, called Rows in ibis.

The API entry point is ibis.rows which returns a Table expression.

The motivation is to allow use of simple in memory datasets in expressions.

ibis.rows handles type inference which makes it suitable for handling the case of inserting into/creating tables from Python objects.

Notes

API Stuff

  1. Iterables of list/tuples/dicts are supported.
  2. ibis.rows has both a columns argument and a schema argument. If schema is None, the function performs type inference on the entire dataset.
  3. Things become unusably slow somewhere between 10k and 200k rows depending on the number of columns. This seems fine to me.
  4. When both columns and schema are None and the row type is not a mapping the default naming scheme is col{i} where i is the zero-indexed ith column number. This template can be changed using an option rows_default_column_name_template.

Backend Specific Stuff

  1. ClickHouse doesn't seem to support the notion of VALUES. It might be posible to implement it by arrayJoin-ing an array of Tuple types. I left that as a follow up.
  2. MySQL and SQLite don't support SELECT-ing from a VALUES statement directly, because SQLAlchemy compiles these incorrectly to a form that these two dialects of SQL don't support. Expressions based on ibis.rows won't work those backends. con.create_table works and is tested for both.
  3. I didn't implement impala support for this, and I won't until someone who isn't a maintainer actively asks for it :)
  4. PySpark translation is implemented.
  5. Pandas and Dask are not implemented, these are in theory straightforward.

cc @tswast

Closes #3832.

cpcloud avatar Aug 08 '22 11:08 cpcloud

Test Results

       6 files         6 suites   3m 50s :stopwatch: 3 184 tests 3 170 :heavy_check_mark: 14 :zzz: 0 :x: 9 552 runs  9 510 :heavy_check_mark: 42 :zzz: 0 :x:

Results for commit 94b17f70.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Aug 08 '22 11:08 github-actions[bot]

Hello @cpcloud! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2022-08-25 10:35:31 UTC

pep8speaks avatar Aug 16 '22 12:08 pep8speaks

Codecov Report

Merging #4323 (5e186ed) into master (9796d4a) will decrease coverage by 0.10%. The diff coverage is 88.58%.

:exclamation: Current head 5e186ed differs from pull request most recent head 94b17f7. Consider uploading reports for the commit 94b17f7 to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4323      +/-   ##
==========================================
- Coverage   92.68%   92.58%   -0.11%     
==========================================
  Files         181      181              
  Lines       20555    20694     +139     
  Branches     2928     2950      +22     
==========================================
+ Hits        19052    19159     +107     
- Misses       1136     1166      +30     
- Partials      367      369       +2     
Impacted Files Coverage Δ
ibis/backends/pyspark/__init__.py 86.00% <ø> (ø)
ibis/expr/schema.py 80.55% <21.73%> (-11.18%) :arrow_down:
ibis/backends/base/sql/compiler/select_builder.py 97.86% <75.00%> (-0.34%) :arrow_down:
ibis/expr/format.py 93.10% <75.00%> (-0.71%) :arrow_down:
ibis/expr/rules.py 88.21% <75.60%> (-1.93%) :arrow_down:
ibis/expr/api.py 97.34% <93.33%> (-0.31%) :arrow_down:
ibis/backends/base/sql/alchemy/__init__.py 90.27% <100.00%> (+0.38%) :arrow_up:
ibis/backends/base/sql/alchemy/query_builder.py 93.44% <100.00%> (+0.10%) :arrow_up:
ibis/backends/base/sql/compiler/query_builder.py 96.72% <100.00%> (+0.03%) :arrow_up:
ibis/backends/duckdb/compiler.py 97.36% <100.00%> (+0.07%) :arrow_up:
... and 7 more

codecov[bot] avatar Aug 24 '22 10:08 codecov[bot]