Where does documentation for `module` stand?

[richb-hanover]

What's up?

I have a set of functions that I would like to include in each of my PRQL query files. I can't make it work.

I attempted to paste one of the definitions of a module from the Modules page into the Playground, but got this error message:

explicit module declarations are not allowed

I also tried simply adding module my_module_file.prql to the beginning of another query in an attempt to import the definitions, but no dice.

What else do I need to know? :-) Many thanks.

Update: This is the set of function declarations I use in several queries... If I'm understanding modules correctly, I could create the file once and simply "name it" in each query...

# dollars displays a numeric value as dollars with commas
let dollars = d -> s"""printf('$%,d',{d})"""

# pct computes the amount (percent) the new differs from old
let pct = o n -> 100.0*( n - o ) / o

# prtpct prints a floating point number with "%"
let prtpct = v -> s"printf('%1.1f%'', {v})"

[max-sixty]

Good question, thanks for asking.

To directly answer — I would say the module docs don't really exist at the moment, and that's partly intentional. While I'm v excited about modules, I'm not sure @aljazerzen has put his stamp on them to say they're ready. (and we've had quite a flow of bug reports recently, so probably that should update us to push newer features less? Or maybe modules are different? Up to @aljazerzen on how confident he is...)

To try them out, check out the example at https://github.com/prql/prql/blob/ef89bba0ca3f35c5b1cc1aa48cf814fe6ed5d4a4/crates/prqlc/tests/project/Project.prql

Note that these don't work in VS Code yet.

[richb-hanover]

OK. Sounds as I should leave modules for a while 'til the implementation matures. Some questions:

• Am I correct that modules will address my use case of including common functions from a separate file?

• I don't see how that example (linked above) demonstrates modules. It looks like a typical use of let. And it seems to work in VS Code. Or am I looking at the wrong example?

• Finally, that example seems to have a typo. The join artists.input ... statement gives this error in the Playground:

  Error: Catalog Error: Table with name input does not exist!
  Did you mean "temp.pg_catalog.pg_type"?
  

Thanks as always.

[max-sixty]

• Am I correct that modules will address my use case of including common functions from a separate file?

Yes!

• I don't see how that example (linked above) demonstrates modules. It looks like a typical use of let. And it seems to work in VS Code. Or am I looking at the wrong example?

Check out https://github.com/PRQL/prql/blob/ef89bba0ca3f35c5b1cc1aa48cf814fe6ed5d4a4/crates/prqlc/tests/project/artists.prql !

-- prqlc compile crates/prqlc/tests/project/
WITH table_1 AS (
  SELECT
    120 AS artist_id,
    DATE '2023-05-18' AS last_listen
  UNION
  ALL
  SELECT
    7 AS artist_id,
    DATE '2023-05-16' AS last_listen
),
favorite_artists AS (
  SELECT
    artist_id,
    last_listen
  FROM
    table_1
),
table_0 AS (
  SELECT
    *
  FROM
    read_parquet('artists.parquet')
),
input AS (
  SELECT
    *
  FROM
    table_0
)
SELECT
  favorite_artists.artist_id,
  favorite_artists.last_listen,
  input.*
FROM
  favorite_artists
  LEFT JOIN input ON favorite_artists.artist_id = input.artist_id

-- Generated by PRQL compiler version:0.9.3 (https://prql-lang.org)

...including read_parquet('artists.parquet')

Does that make sense?

[richb-hanover]

I'm sorry to say that this doesn't help me to understand how modules work. The current documentation seems to have a lot of information that's meaningful for people with a deep understanding and familiarity with PRQL, but is hard for newcomers to interpret.

I'm looking to bridge that gap for newcomers by learning enough to be able to explain what PRQL actually does while retaining a nod to the deep principles behind the feature.

Perhaps I have difficulty because the documentation currently is confusing/contradictory. For example, it says:

This section is under discussion. Current implementation plans do not include module declarations, but loading of all files under the compilation path.

I'm not certain about the full import of that statement. And then...

To include PRQL source code from other files, we can use the following syntax:

module my_playlists

This makes it seem as if I should be able to enter module my_playlists in my query - but that generates an error message. That page also shows a deeply nested module declaration that isn't fully fleshed out and can't be pasted into the Playground for experimentation.

I'm completely comfortable living with the understanding that modules aren't quite ripe. I can do useful work without them. And when they do mature, I'll check the then-current documentation to test out my understanding

[max-sixty]

Yeah, that documentation is not helpful for that goal at the moment, sorry if that wasn't clearer.

Let's keep this open and we can come back to it when modules are further along...

[richb-hanover]

Thanks

[richb-hanover]

@max-sixty @aljazerzen Now that 0.10.0 has shipped, when would be a good time to review the status of modules?

I have seen messages (I think) stating that prqlc will handle module inclusion, but I couldn't tell whether that would help a newcomer using the Playground or VS Code extension... Thanks as always.

[max-sixty]

Yes!

To some extent this involves us getting more comfortable with its design. I think that currently we could definitely add some more docs even if the design might change a bit. Though will obv update on @aljazerzen 's view...

(sorry I've been offline for a few days, and slow on contributions for a few weeks, I'm intending to be back at full pace over the coming weeks)

[richb-hanover]

@max-sixty @aljazerzen I'm looking for an update on the documentation for modules.

I am creating a bunch of queries that use a lot of domain-specific functions and derive/case statements. I'd like to avoid copy/pasting them into each query.

Is this capability available in PRQL now? Or should I keep the old copy/paste process going for a while? Thanks.

[max-sixty]

No update at the moment — please feel free to explore what's there at the moment — and now I'm back online I'll try and get to this at some point. But we're currently going through a rethink of some areas such as the resolver, so this isn't quite at the front of the queue...

[richb-hanover]

No apologies necessary for taking time away from PRQL.

However, I think I'm not understanding something... When I read File Importing I see the comment "This section is under discussion..." However, that text also makes it seem as if the line below would work. These seem contradictory.

And in fact, when I use this:

module useful_functions               

I see this error code:

Error: 
    ╭─[:14:24]
    │
 14 │ module useful_functions
    │                        ┬  
    │                        ╰── module definition expected {, but found new line
────╯

Thoughts? Thanks!

[max-sixty]

I think that's what the Current implementation plans do not include module declarations, but loading of all files under the compilation path. caveat means, but feel free to make it clearer that it's not yet supported...

[richb-hanover]

Ahah... The importance of that statement wasn't clear to me. I think I was led astray by the several sections on that page that appear to show sensible examples.

Would it be correct to say this at the top of the page, and have a strong warning above each of the examples?

The notes below document our understanding of the way the final PRQL compiler will likely work. The PRQL compiler currently uses these techniques to build the standard std, date, text, andmath modules into the language.

However, at this time, none of the examples below work from a PRQL query iteself. That is, the module statement cannot import files from the local file system.

Many thanks

[max-sixty]

Would it be correct to say this at the top of the page, and have a strong warning above each of the examples?

Yes, particularly strengthening the warning!