vulcan-sql icon indicating copy to clipboard operation
vulcan-sql copied to clipboard

Support Postgres exporting results to Parquet directly

Open kokokuo opened this issue 2 years ago • 0 comments

What’s the problem you're trying to solve

In #150, In order to enhance our query performance after users send the API request to run our data endpoint to get the result from the data source. We need to provide a Caching (pre-loading) Layer with the duckDB to enhance query performance.

Describe the solution you’d like

According to #153, we should make sure the DataSource have export method contract or not, if not please add it to make your PGDataSource could implement it.

export interface ExportOptions {
  // The sql query result to export
  sql: string;
  // The directory to export result to file
  directory: string;
  // The profile name to select to export data
  profileName: string;
  // export file format type
  type: CacheLayerStoreFormatType | string;
}

..... 

@VulcanExtension(TYPES.Extension_DataSource, { enforcedId: true })
export abstract class DataSource<
  C = any,
  PROFILE = Record<string, any>
> extends ExtensionBase<C> {
  private profiles: Map<string, Profile<PROFILE>>;

  constructor(
    @inject(TYPES.ExtensionConfig) config: C,
    @inject(TYPES.ExtensionName) moduleName: string,
    @multiInject(TYPES.Profile) @optional() profiles: Profile[] = []
  ) {
    super(config, moduleName);
    this.profiles = profiles.reduce(
      (prev, curr) => prev.set(curr.name, curr),
      new Map()
    );
  }
  ....

  /**
   * Export query result data to cache file for cache layer loader used
   */
  public export(options: ExportOptions): Promise<void> {
    throw new Error(`Export method not implemented`);
  }

  .....
}

Additional Context

The PostgreSQL may need to search how to export the result directly to the parquet file.

kokokuo avatar Apr 06 '23 07:04 kokokuo