Feature/speedup parquet export

Open jcharkow opened this issue 1 year ago • 1 comments

Re-implementation of PR https://github.com/PyProphet/pyprophet/pull/111

This uses duckdb for parquet export instead of pandas. This allows for memory and SQL queries to be handled by duckdb which is faster and uses less memory.

Oct 09 '24 17:10 jcharkow

Currently, this method only supports a combined output file (no split based on runs) and no IPF

Oct 09 '24 18:10 jcharkow

@singjc can you review this when you get a chance?

Oct 09 '24 18:10 jcharkow

Looks great, thank you! Please feel free to merge when you think that it is ready.

Oct 09 '24 18:10 grosenberger

@singjc suggestions should be addressed now

Oct 09 '24 21:10 jcharkow