seafowl

seafowl copied to clipboard

Figure out what to do with `table_column` catalog table and bulk schema loading in general

1

Currently we're not really using our [`Schema`](https://github.com/splitgraph/seafowl/blob/40b1158a90121422e66acbc66e4d536f6081b6d7/src/schema.rs#L10-L13) for anything but the `to_column_names_types` call when persisting the columns to the `table_column` metadata table. So it's possible to remove that `Schema` altogether...

gruuya

Currently we don't support these, since we have a custom delta Parquet writer (i.e. `plan_to_object_store`) as a way to mitigate memory performance of writes. This could be achieved in two...

gruuya

Remote tables: filter without sort

2

I was wondering: does the `datafusion_remote_tables` filter push-down not support sorting? It seems that using filters and limits in the absence of a sort order could lead to un-expected results....

backkem

`datafusion-remote-tables` crate

4

According to the blog post [here](https://www.splitgraph.com/blog/datafusion-filter-expr-visitor), `datafusion-remote-tables` was released separately as a crate, but I don't see it on `crates.io`

ccollie

feat: iceberg support

1

- any plan to support the iceberg format instead of the delta format

terekete

Arrow JSON writer that we use to format the output result of queries [does not support](https://github.com/apache/arrow-rs/blob/master/arrow-json/src/writer.rs#L380-L385) all data types that Datafusion handles. For example: ```bash $ curl -H "Content-Type: application/json"...

gruuya

Bumps [vergen](https://github.com/rustyhorde/vergen) from 7.5.1 to 8.2.3. Release notes Sourced from vergen's releases. v8.2.3 Updated gix to 0.48.0 Added support for specifying a custom build.rs location. Added support for using a...

dependabot[bot]

Bumps [clap](https://github.com/clap-rs/clap) from 3.2.25 to 4.3.10. Release notes Sourced from clap's releases. v4.3.10 [4.3.10] - 2023-06-30 Performance Drop a dependency, reducing binary size by 1.3 KiB v4.3.9 [4.3.9] - 2023-06-28...

dependabot[bot]

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.5 to 0.11.0. Changelog Sourced from itertools's changelog. 0.11.0 Breaking Make Itertools::merge_join_by also accept functions returning bool (#704) Implement PeekingNext transitively over mutable references (#643) Change with_position...

dependabot[bot]

Bumps [arrow-csv](https://github.com/apache/arrow-rs) from 40.0.0 to 42.0.0. Changelog Sourced from arrow-csv's changelog. Historical Changelog 41.0.0 (2023-06-02) Full Changelog Breaking changes: Rename list contains kernels to in_list (#4289) #4342 [parquet] [arrow] (tustvold)...

dependabot[bot]