quackpipe
quackpipe copied to clipboard
Parse and process MergeTree CREATE TABLE expression
What
Desired functionality:
- A user makes a create table request:
POST /qp/query
...
create_table: experiment
fields:
a: UInt64
b: String
c: Float64
engine: Merge
order_by:
- a
timestamp:
field: a
precision: ms
partition_by:
- b
Or the json request with the same data.
- The processing component should create the following entry int the
tables
table:
name: experimental,
path /tmp/experimental,
field_names ['a', 'b', 'c'],
field_types ['UInt64', 'String', 'Float64'],
order_by ['a'],
engine 'Merge',
timestamp_field: 'a',
timestamp_precision: 'ns',
partition_by: ['b'],
- The processing component should create the
/tmp/experimental/tmp
and/tmp/experimental/data
folders.
#How
- Create an endpoint
/qp/query
. - Add the following checks to /qp/query endpoint
- Content-type should be either yaml or json
- If the decoded structure doesn't have "create_table" attribute, return 400 - Not supported error
- Check if the decoded structure has all the attributes and the attributes are of the desired type
- Check if the name of the table and names of the fields are alphanumeric or
_
character and don't start with a number. - Check if the fields enumerated in order_by, timestamp, partition_by are present in the
fields
attributes.
- If all the checks are passed, the system should create an entry in the
tables
table and create the folders on the HD.