Subscribe for updates

[max-sixty]

trafficstars

I'm hoping to bring prql from a proposal to reality. If you'd like updates in your GH notifications, subscribe to this issue by hitting the button on the right.

[max-sixty]

Update 2022-01-27

Thanks to everyone for the enthusiasm and support over the past couple of days! It seems that folks are similarly frustrated by SQL, and believe we can build something better.

We've already made some improvements to the language, based on dozens of folks' feedback, both here and on HackerNews. I think the basics are in a good place now, and there will inevitably be more refinements.

I just merged an initial roadmap. The immediate focus is to start building the parser, so check out #26. For folks who are up for writing some (fairly simple) rust, that's the highest impact contribution you can make to the project at the moment. We had our first PR on this from @iafisher yesterday (#30).

I asked for community ideas in #33, comment there if you have thoughts.

Thank you! Max

[max-sixty]

Update 2022-02-08

We've come quite a long way in two weeks! We now have:

• A parser that parses most of the examples. Though a few big things still to do: #26. Thanks to @iafisher for adding code to parse functions.
• The beginnings of "stage 1 transpiler", which modifies the AST; we just merged code that replaces variables in a pipeline's AST.
• A good set of tests & examples. Thanks to @danjones1618 for setting up CI.

Next up:

• The beginning of "stage 2 transpiler", which writes SQL from an AST.
• Complete the "stage 1 transpiler", which will run functions. Functions won't be recursive, and so can be run in turn, replacing one set of AST Items with another. Thanks to @jamii for the discussion.
• Complete the remaining items of the parser.

The discord has been good to discuss contributions and the language — come join if you think you might be up for contributing.

[max-sixty]

Update 2022-03-18

Getting to v0.1

We're most of the way to a working version that people can use for experimentation!

• We can fully transpile queries such as the Readme's first example (pasted below), and most of the Readme's second example.
• We can parse all queries.
• We can "materialize" functions & variables.
• We can "translate" most of the query to SQL. We're using sqlparser-rs, which means we only need to convert to its AST, and it will do the writing.
• I just updated the Roadmap a few days ago — we're a handful of big issues away from v0.1; probably 80% of the way there. My guess would be that it's few™ weeks away, depending on my weekend capacity, other contributions, and whether we hit unexpected bumps.

PyPrql

@qorrect has built a whole version of Prql in Python — https://github.com/qorrect/PyPrql — which is impressively complete, including even stretching to some of the items in "Fast feedback" in the roadmap, like an autocomplete that's cohesive with a database.

I'd encourage people to check it out, and contribute, particularly if they're more comfortable with Python.

Notable contributions

• @kwigley made a whole CLI. Now no rust is needed to use PRQL.
• @kiibo382 added parsing for both Tables and Joins

How you can help

I added some more detail to this section on the Readme. In particular, small contributions are very welcome — post an example of something not working, add a test, add a docstring, etc. Contributing helps the project both from the code you contribute, and as a broader signal to the community that people are excited about the project.

The Discord is lively, with conversations most days.

If there's anything stopping you contributing (confusing code, unclear where to start, dev environment, etc), please let me know, I will do everything I can to fix it.

Example of what now works

Copying the results of the link to the first example here:

From

from employees
filter country = "USA"                           # Each line transforms the previous result.
derive [                                         # This adds columns / variables.
  gross_salary: salary + payroll_tax,
  gross_cost:   gross_salary + benefits_cost     # Variables can use other variables.
]
filter gross_cost > 0
aggregate by:[title, country] [                  # `by` are the columns to group by.
    average salary,                              # These are aggregation calcs run on each group.
    sum     salary,
    average gross_salary,
    sum     gross_salary,
    average gross_cost,
    sum_gross_cost: sum gross_cost,
    count: count,
]
sort sum_gross_cost
filter count > 200
take 20

to SQL:

SELECT
    TOP (20) salary + payroll_tax AS gross_salary,
    salary + payroll_tax + benefits_cost AS gross_cost,
    SUM(salary + payroll_tax + benefits_cost) AS sum_gross_cost,
    COUNT(*) AS count,
    AVG(salary),
    SUM(salary),
    AVG(salary + payroll_tax),
    SUM(salary + payroll_tax),
    AVG(salary + payroll_tax + benefits_cost),
    *
FROM
    employees
WHERE
    country = 'USA'
    and salary + payroll_tax + benefits_cost > 0
GROUP BY
    title,
    country
HAVING
    COUNT(*) > 200
ORDER BY
    SUM(salary + payroll_tax + benefits_cost)

[max-sixty]

Update 2022-04-01

We released 0.1.0 last week, and just released 0.1.1! So now it's possible to try PRQL: cargo install prql

It's still early, but it's now easy to try queries out and see their result with prql compile. See an example below

We've also laid out the vision for the next big milestone, including an interactive compiler in a browser, better documentation, and more language features.

If you're interesting in joining, check out the Contributing section in the Readme — we'd love to have you as part of the team!

---

echo 'from employees
  filter country = "USA"                           # Each line transforms the previous result.
  derive [                                         # This adds columns / variables.
    gross_salary: salary + payroll_tax,
    gross_cost:   gross_salary + benefits_cost     # Variables can use other variables.
  ]
  filter gross_cost > 0
  aggregate by:[title, country] [                  # `by` are the columns to group by.
      average salary,                              # These are aggregation calcs run on each group.
      sum     salary,
      average gross_salary,
      sum     gross_salary,
      average gross_cost,
      sum_gross_cost: sum gross_cost,
      ct: count,
  ]
  sort sum_gross_cost
  filter ct > 200
  take 20' | prql compile

SELECT
  TOP (20) title,
  country,
  AVG(salary),
  SUM(salary),
  AVG(salary + payroll_tax),
  SUM(salary + payroll_tax),
  AVG(salary + payroll_tax + benefits_cost),
  SUM(salary + payroll_tax + benefits_cost) AS sum_gross_cost,
  COUNT(*) AS ct
FROM
  employees
WHERE
  country = 'USA'
  AND salary + payroll_tax + benefits_cost > 0
GROUP BY
  title,
  country
HAVING
  COUNT(*) > 200
ORDER BY
  sum_gross_cost

[max-sixty]

Update 2022-06-27

It's been a while since the last update, but not for a lack of progress — today we're releasing 0.2 — the first version you can use in your own projects.

We just posted this to HackerNews and Reddit

Let us know your feedback!

[max-sixty]

Update 2022-07-10

We have a small patch release out, with the result of some excellent bug reports, fixes, and internal improvements: https://github.com/prql/prql/releases/tag/0.2.2

[max-sixty]

Hi everyone!

As you can see, I've been bad at updating this issue. Sorry for falling short of expectations here.

This is a communication failure rather than a lack of progress! The project has been doing better than we could have hoped for — https://github.com/PRQL/prql/releases has the log of recent changes.

For a project summary, I just added a monthly update in https://github.com/PRQL/prql/pull/2257, and I may start including these in the Release notes.

So — instead of following this issue, follow the releases, which have been coming with nice Changelogs for the past few months. Here's how to follow releases.

Thanks for subscribing, I really hope you'll continue participating in the project.

Max