pypsa-usa icon indicating copy to clipboard operation
pypsa-usa copied to clipboard

Change processing of EFS data to xarray

Open trevorb1 opened this issue 1 year ago • 1 comments

Feature Request

With the introduction of ReadEfs in PR #202, more EFS data is retained for sector coupling studies. The issue is, if many sectors are using the EFS data and try to process it at the same time, (I think) memory consumption skyrockets as I am sometimes (and seemingly randomly) getting killed errors when running. This has not been an issue for electricity only studies or if running the script by itself.

Suggested Solution

I believe xarray will be much more memory efficient at processing EFS data.

Additional Info

No response

trevorb1 avatar Apr 22 '24 00:04 trevorb1

One way we could significantly reduce data size significantly would be to aggregate to substations earlier on in the workflow. Right now bus:sub is a many:1 ratio where all buses have the same GPS location based on the the substation. This would involve making many modifications throughout the workflow, so no quick task. We would probably want to change build_base_network to aggregate bus to sub.

ktehranchi avatar Apr 22 '24 17:04 ktehranchi