activitysim icon indicating copy to clipboard operation
activitysim copied to clipboard

Support reading skims in Parquet format

Open amsamimi opened this issue 3 months ago • 6 comments

Description: Currently, skims are read from OMX files, but based on our experience, storing skims in Parquet format can reduce file size by ~30%. This would significantly reduce storage requirements and improve efficiency. We already support reading Parquet instead of CSV for the population synthesizer inputs, which has proven very valuable for reducing the size of model inputs. Extending this capability to skims would provide similar benefits.

Proposed enhancement: Add support for reading skims directly from Parquet format. Maintain backward compatibility with existing OMX inputs.

Benefits: Reduced disk usage. Faster I/O performance.

amsamimi avatar Aug 26 '25 23:08 amsamimi