quackpipe icon indicating copy to clipboard operation
quackpipe copied to clipboard

Parse and process MergeTree CREATE TABLE expression

Open akvlad opened this issue 6 months ago • 1 comments

What

Desired functionality:

  1. A user makes a create table request:
POST /qp/query
...
create_table: experiment
fields:
  a: UInt64
  b: String
  c: Float64
engine: Merge
order_by: 
  - a
timestamp:
  field: a
  precision: ms
partition_by:
  - b

Or the json request with the same data.

  1. The processing component should create the following entry int the tables table:
  name: experimental, 
  path /tmp/experimental,
  field_names ['a', 'b', 'c'],
  field_types ['UInt64', 'String', 'Float64'],
  order_by ['a'],
  engine 'Merge',
  timestamp_field: 'a',
  timestamp_precision: 'ns',
  partition_by: ['b'],
  1. The processing component should create the /tmp/experimental/tmp and /tmp/experimental/data folders.

#How

  1. Create an endpoint /qp/query .
  2. Add the following checks to /qp/query endpoint
  • Content-type should be either yaml or json
  • If the decoded structure doesn't have "create_table" attribute, return 400 - Not supported error
  • Check if the decoded structure has all the attributes and the attributes are of the desired type
  • Check if the name of the table and names of the fields are alphanumeric or _ character and don't start with a number.
  • Check if the fields enumerated in order_by, timestamp, partition_by are present in the fields attributes.
  1. If all the checks are passed, the system should create an entry in the tables table and create the folders on the HD.

akvlad avatar Aug 13 '24 10:08 akvlad