pg_analytics icon indicating copy to clipboard operation
pg_analytics copied to clipboard

test: Support for Multi-Level Partition Tables

Open shamb0 opened this issue 6 months ago • 3 comments

What

Implements a demonstration test for multi-level partition tables, addressing issue #56.

Why

This demo showcases the pg_analytics extension's capability to support multi-level partitioned tables. The implementation organizes data hierarchically, enabling efficient access to context-relevant information.

How

  1. Generates a simulated Auto Sales dataset and creates a local Parquet data source.
  2. Utilizes DataFusion to load sales data from the Parquet file into a DataFrame.
  3. Partitions the data and uploads partitions to an S3 bucket.
  4. Configures necessary tables and pg_analytics Foreign Data Wrapper (FDW) in PostgreSQL using S3 data.
  5. Executes analytics test queries on the multi-level partitioned table.

Tests

To run the demonstration test:

RUST_LOG=info \
    cargo test \
    --test \
    test_mlp_auto_sales \
    -- \
    --nocapture

Test traces are available in the attached log file :: https://gist.githubusercontent.com/shamb0/2ed909ac9604c610af1d7fa0e87f9a82/raw/02a4203cdc1d675181d9f9700c578c81405becdb/wk2434-pg_analytics-mlp-demo.md

shamb0 avatar Aug 20 '24 05:08 shamb0