quackpipe icon indicating copy to clipboard operation
quackpipe copied to clipboard

Create DuckDB internal schema

Open akvlad opened this issue 6 months ago • 1 comments

What

Every quackpipe instance should have an internal tables describing the created "merged parquet" tables.

The tables should contain

  • table name
  • table base folder path on HD
  • table fields and types
  • table order by expression

How

Add the create table logic on quackpipe startup:

CREATE TABLE tables (
  name VARCHAR PRIMARY KEY, 
  path VARCHAR,
  field_names VARCHAR[],
  field_types VARCHAR[],
  order_by VARCHAR[],
  engine VARCHAR[],
  timestamp_field: VARCHAR[],
  timestamp_precision: VARCHAR[],
  partition_by: VARCHAR[],
);

The table should be created on startup if it does not exist

The table should persist between restarts

Test the logic:

  • start a quackpipe instance. Table should be created
  • insert some data into the table
  • restart the instance. The table should contain the inserted data

akvlad avatar Aug 13 '24 10:08 akvlad