pudl
pudl copied to clipboard
Handle high memory usage in CEMS parquet-to-duckdb conversion
Overview
Details here: https://github.com/catalyst-cooperative/pudl/pull/3741#issuecomment-2257443287
But basically, the CEMS parquet file is pretty big and that's causing a memory issue when trying to convert to DuckDB.
Success Criteria
- [ ] Hourly CEMS makes it into DuckDB
### Next steps
* [ ] figure out if we can convert the parquet file by chunks