wren-engine icon indicating copy to clipboard operation
wren-engine copied to clipboard

Support Bi boost

Open liun03 opened this issue 2 years ago • 2 comments

BigQuery

Syntax and data

Metric tables:

  • [x] parser: make sure the statement includes metric only. https://github.com/Canner/canner-metric-layer/pull/214
  • [x] sql rewrite: rewrite statement and query duckdb with pre-aggregation table https://github.com/Canner/canner-metric-layer/pull/214
  • [x] 🔥check return data format or types
  • [x] exclude unsupported syntax which can be executed in duckdb,e.g explain, show, describe, insert, create. https://github.com/Canner/canner-metric-layer/pull/214
  • [x] provide a workaround for https://github.com/duckdb/duckdb/issues/5673

Storage:

  • [x] export data to parquet: https://github.com/Canner/canner-metric-layer/pull/206
  • [x] import data to duckdb: https://github.com/Canner/canner-metric-layer/pull/206
  • refresh:
  • [x] 🔥fix rate
  • [ ] expression

Canner

pre-condition

can used in canner

Storage:

  • [x] *🔥canner provide export parquet api: provide APIs to export parquets to cloud storage and duckdb support s3 and gcs only.
  • [x] *🔥cleaner: canner can clear temp parquets or provide APIs which can be used by accio
  • [ ] *🔥 sync result: wire protocol used
  • [x] *🔥 async result: web team sql explorer?
  • [x] 🔥 adjust refreshing codes

🔥 implementation of export and import data from canner

  • [x] 🔥export data from canner
  • [x] 🔥import data to duckdb: file base; ~s3, azure blob,~ local (no gcs in canner)

note: * implemente in canner ; 🔥 critical

Others

  • [ ] memory management
  • [x] 🔥extract module for bi boost

liun03 avatar Mar 29 '23 09:03 liun03

https://github.com/duckdb/duckdb/pull/8302 has been merge

cc @liun03 @wwwy3y3 @brandboat

douenergy avatar Jul 20 '23 14:07 douenergy

https://github.com/duckdb/duckdb/pull/8302 has been merge

Many thanks !!!

brandboat avatar Jul 20 '23 14:07 brandboat