Andrew Lamb
Andrew Lamb
Here is a PR that attempts to clarify the current status (does not attempt to actually define the ordering): - https://github.com/apache/parquet-format/pull/504
Here is a document with background clarification as well: - https://docs.google.com/document/d/1Ox0qHYBgs_3-pNqn9V8zVQm_W6qP0lsbd2XwQnQVz1Y/edit?tab=t.0#heading=h.nx463wi3cktx
In my opinion, adding new index-like structures to the parquet spec makes sense when a "large" number of engines will support writing and using them. Today it is possible to...
I believe @emkornfield expresses similar sentiments in his response on the mailing list: https://lists.apache.org/thread/r2xfqk9kx974hhh23zr06jy80dvlhnmd
> Today it is possible to use such index structures without changing the spec in at least two ways: I wrote some blogs about how this process works. You can...
i can probably help with the page index changes
I wonder if @jorgecarleitao @ritchie46 @b41sh or others can share their perspectives on this propsal
Specifically, perhaps we can help show how to use `parquet` (rather than `parquet2`) in polars using this new interop -- and then let any other migration take its course
It appears that databend, another of the historical major users of arrow2 has also switched to arrow-rs https://github.com/search?q=repo%3Adatafuselabs%2Fdatabend+arrow+language%3ATOML&type=code&l=TOML
> Maybe it makes sense to add a note to the readme of this repo explaining the status as well? Here is a proposal to add a note to the...