quackpipe
quackpipe copied to clipboard
Create DuckDB internal schema
What
Every quackpipe instance should have an internal tables describing the created "merged parquet" tables.
The tables should contain
- table name
- table base folder path on HD
- table fields and types
- table
order by
expression
How
Add the create table logic on quackpipe startup:
CREATE TABLE tables (
name VARCHAR PRIMARY KEY,
path VARCHAR,
field_names VARCHAR[],
field_types VARCHAR[],
order_by VARCHAR[],
engine VARCHAR[],
timestamp_field: VARCHAR[],
timestamp_precision: VARCHAR[],
partition_by: VARCHAR[],
);
The table should be created on startup if it does not exist
The table should persist between restarts
Test the logic:
- start a quackpipe instance. Table should be created
- insert some data into the table
- restart the instance. The table should contain the inserted data