pudl icon indicating copy to clipboard operation
pudl copied to clipboard

Handle high memory usage in CEMS parquet-to-duckdb conversion

Open jdangerx opened this issue 6 months ago • 0 comments

Overview

Details here: https://github.com/catalyst-cooperative/pudl/pull/3741#issuecomment-2257443287

But basically, the CEMS parquet file is pretty big and that's causing a memory issue when trying to convert to DuckDB.

Success Criteria

  • [ ] Hourly CEMS makes it into DuckDB
### Next steps
* [ ] figure out if we can convert the parquet file by chunks

jdangerx avatar Aug 26 '24 13:08 jdangerx