prql icon indicating copy to clipboard operation
prql copied to clipboard

feat: select exclude

Open aljazerzen opened this issue 2 years ago • 4 comments

Just opening this PR to give advice on how to do this

aljazerzen avatar Dec 23 '22 10:12 aljazerzen

https://github.com/PRQL/prql/pull/1329/files#diff-d0a5d98a5eeaa51d37d7fbb753111872d244f7586131dda4fcf35c5d1f071426R14-R17 This may cause problems / will have to be handled in quite a few places.

Instead I suggest leveraging this: https://github.com/prql/prql/blob/0dbb4b5e24e6990e56db0157a8447a2d5f130fe9/prql-compiler/src/ast/rq/mod.rs#L52-L53

If we declare all mentioned column in the ExternTable relation, the wildcard will now mean "all other columns". That combined with other known columns not mentioned in the exclude, will achieve the select we want.

aljazerzen avatar Dec 23 '22 10:12 aljazerzen

For example:

from albums          # at this point we know only of [albums.*]
                     # (this wildcard means "there are other columns we don't know about)
filter album_id > 10 # here we know [albums.*, albums.album_id]
select -[title]      # and now we   [albums.*, albums.album_id, albums.title]

For this, we'd emit RQ looking like:

tables: 
  - id: 0
    name: albums
    columns: [Wildcard, album_id, title]
    relation: ...
relation:
  Pipeline:
  - From:
      source: 0
      columns: [/*wildcard*/1, /*album_id*/2, /*title*/3]
  - Filter:
      ...
  - Select: [1, 2] # excludes title

aljazerzen avatar Dec 23 '22 10:12 aljazerzen

Translator will try to exclude the columns, but in this last example, it won't be able to and will just report a warning here: https://github.com/prql/prql/blob/0dbb4b5e24e6990e56db0157a8447a2d5f130fe9/prql-compiler/src/sql/translator.rs#L492

So this should be extended to either:

  • emit EXCLUDE if dialects support it,
  • emit an error, saying "need more information, consider adding a select that lists all columns right after from"

aljazerzen avatar Dec 23 '22 11:12 aljazerzen

Awesome, thanks a lot for adding the comments — you probably saw that I had pushed this as a quick experiment to see if I could continue moving through the compiler.

This comment is v clear — thanks, I can visualize that working nicely.

max-sixty avatar Dec 28 '22 00:12 max-sixty