seafowl
seafowl copied to clipboard
splitgraph
Analytical database for data-driven Web applications 🪶
We currently do not support UDAFs (user defined aggregation functions), even though DataFusion does (https://docs.rs/datafusion/latest/datafusion/physical_plan/udaf/struct.AggregateUDF.html). The most basic implementation would be expecting the WASM function to be an "accumulator" (which...
Done in https://github.com/splitgraph/seafowl/pull/71: ~~- `MemoryManagerConfig`: max runtime memory usage for plan execution (rough, since it doesn't track basic process data structures): https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/memory_manager.rs#L35-L55 / https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/runtime_env.rs#L141-L145~~ ~~- `DiskManagerConfig`: using the OS temp...
Add ability to cache query results in the same object storage that we use for actual Parquet files. This might not be crucial if we implement https://github.com/splitgraph/seafowl/issues/20 (in which case...
(followup to https://github.com/splitgraph/seafowl/issues/20) The current default is not sending any cache-control headers at all, which means that the browser/CDNs will lean towards more caching. (from my basic testing, it seems...
If the GET API receives an unhashed SQL query in the URL (e.g. just urlencoded), hash it and redirect the user to the hashed endpoint), as per https://github.com/splitgraph/seafowl/issues/20#issuecomment-1204130690 Need to...
When using an S3-compatible object store, DataFusion / Seafowl currently download the object every time they need to scan through it. We should let users dedicate some scratch space on...
Currently, when writing data: - we create a region for every partition of the original plan (a row in the `physical_region` table in the database) - the region has a...
https://github.com/splitgraph/seafowl/blob/main/src/context.rs#L1090-L1091 Currently, the Parquet file doesn't seem to store min-max values for utf-8 and probably other (like timestamp) columns.
https://github.com/splitgraph/seafowl/blob/f00efc451aaa80a818b42e5d0be72efe39f3f50c/migrations/20220705072248_create_tables.up.sql Make sure that, when we need to, we can migrate: - the table storage format (e.g. tag a table with "v1" and let users upgrade the table) - region...
Metadata
Owner
Metadata
Analytical database for data-driven Web applications 🪶