vulcan-sql
vulcan-sql copied to clipboard
Support Postgres exporting results to Parquet directly
What’s the problem you're trying to solve
In #150, In order to enhance our query performance after users send the API request to run our data endpoint to get the result from the data source. We need to provide a Caching (pre-loading) Layer with the duckDB to enhance query performance.
Describe the solution you’d like
According to #153, we should make sure the DataSource have export method contract or not, if not please add it to make your PGDataSource could implement it.
export interface ExportOptions {
// The sql query result to export
sql: string;
// The directory to export result to file
directory: string;
// The profile name to select to export data
profileName: string;
// export file format type
type: CacheLayerStoreFormatType | string;
}
.....
@VulcanExtension(TYPES.Extension_DataSource, { enforcedId: true })
export abstract class DataSource<
C = any,
PROFILE = Record<string, any>
> extends ExtensionBase<C> {
private profiles: Map<string, Profile<PROFILE>>;
constructor(
@inject(TYPES.ExtensionConfig) config: C,
@inject(TYPES.ExtensionName) moduleName: string,
@multiInject(TYPES.Profile) @optional() profiles: Profile[] = []
) {
super(config, moduleName);
this.profiles = profiles.reduce(
(prev, curr) => prev.set(curr.name, curr),
new Map()
);
}
....
/**
* Export query result data to cache file for cache layer loader used
*/
public export(options: ExportOptions): Promise<void> {
throw new Error(`Export method not implemented`);
}
.....
}
Additional Context
The PostgreSQL may need to search how to export the result directly to the parquet file.