PyPSA
Make model instance conveniently inspectable (separate problem prep and solving)
Changes proposed in this Pull Request
This PR refactors solve_network.py (and solve_operations_network.py) with the aim to make an instance of a optimization problem to be solved in the workflow easily inspectable.
Why this PR
The reason is that solve_network function included all steps from model creation, unpacking extra_functionality via kwargs, and solving models based on if-conditions for different optimization models:
https://github.com/PyPSA/pypsa-eur/blob/822a92729e6973aa3aff741d6c94f1da2c75e8b2/scripts/solve_network.py#L1276-L1398
It was therefore inconvenient to investigate model instance (are all required constraints attached? what is the RHS of a specific constraint? etc) I spoke recently with @fneum @bobbyxng @cpschau who all confirmed it is pretty annoying and we should have refactored it long ago. I also ping @lindnemi with whom we discussed this problem long ago.
Similar impression I got from a series of PyPSA(-Eur) workshops we did this year, where folks from industry told me it is not straightforward to investigate optimization problem at hand with PyPSA-Eur.
And finally, I'm on making PyPSA-Eur workflow compatible with https://github.com/PyPSA/PyPSA/pull/1345 and https://github.com/PyPSA/PyPSA/pull/1154 and this PR is a predecessor.
Changes
- I create a dedicated
prepare_optimization_problem()function - and a dedicated
collect_kwargs()that separates model creation args and solver args - and refactor main block to explicit steps incl. prepare_network, prepare_optimization_problem() and solve problem depending on optimization mode
- I keep all model- and solve-related functionality/options from original code
This makes code modular. One can mock_snakemake into a step after model prep and inspect optimization problem instance before solving it. I also move rolling horizon logic fully to a correct script (solve_operations_network.py).
Tests
I tested the workflow for three modes
- [x] snakemake -call results/test-sector-overnight/networks/base_s_5_elec_.nc --configfile config/test/config.overnight.yaml
- [x] snakemake -call results/test-rolling-horizon/networks/base_s_5_elec__op.nc --configfile config/examples/config.rolling_horizon.yaml
- [x] snakemake -call results/test-iterative/networks/base_s_5_elec_.nc --configfile config/examples/config.iterative.yaml
I add config.iterative.yaml and config.rolling_horizon.yaml into collection of example configs within PR as well.
Other
I am aware of major refactoring effort in the draft PR https://github.com/PyPSA/pypsa-eur/pull/1838 and I believe these are complementary. I would prefer we merge this now so I can checkout to work on stochastic-compatible workflow; and this PR can be merged into #1838 during rebase to master.
Checklist
- [x] I tested my contribution locally and it works as intended.
- [x] Code and workflow changes are sufficiently documented.
- [x] A release note
doc/release_notes.rstis added.
Hey @Irieo, this makes sense! happy to review (hope this week). I am just wondering about the bigger picture about how operational optimization could be generalized in to solve_network.py and if we shouldn't keep that flexibility. with outsourcing rolling horizon the workflow would not allows us to directly run the rh for a brownfield network?
Hey @Irieo, this makes sense! happy to review (hope this week). I am just wondering about the bigger picture about how operational optimization could be generalized in to
solve_network.pyand if we shouldn't keep that flexibility. with outsourcing rolling horizon the workflow would not allows us to directly run the rh for a brownfield network?
I agree, operational optimization could be generalised within solve_network (while keeping advantages of this PR for making folks able to inspect model instance before RH run). I refactor here solve_operations_network.py mostly to stick to current structure w/o touching bigger picture. Can volunteer to merge this PR later to #1838 when it will be rebased; as well as merge RH logic into solve_network.py.
upd: ah @FabianHofmann thanks I got what you meant. Indeed, with the implementation we had here, brownfield + RH did run through w/o break but it was single-pass optimization. Now I enabled it. So config/examples/config.myopic-rolling.yaml this would run brownfield fleet and RH as intended.
Looks good and conceptually I agree. There is some small optimization potential in dealing with the kwargs.
I am against dumping example configs (potentially high-maintenance and not tested to ensure they are up to date, lots of boilerplate settings for test case).
Thanks for reviewing it. I agree with the suggestion on kwargs and implemented it now.
WRT example configs: I agree that dumping configs that require maintenance isn't a good practice. However, we currently have no example config (let alone anything under test/) for rolling horizon. The model offers rolling-horizon optimization for both solve_operational_network (dispatch only) and solve_network (inv -> freeze inv -> pass to operational), and none of this is tested. Right now, we would learn about things breaking with RH through a GitHub issue, which isn't great. More, anyone trying to test this functionality (like I am doing here) has to inspect the code to assemble a config, and may run into untested non-working parts. I initially added RH to the test suite, then moved to the examples after some feedback. I'm removing the configs now since they are unrelated to this PR, but I suggest adding a copy of RH config with settings matching the Belgian test case to the test suite. I'm happy to volunteer implementing a minimalistic good enough test.
I have tested this final version of PR with
- overnight foresight (config.overnight.yaml)
- myopic foresight (config.myopic.yaml)
- iterative transmission expansion (config.iterative.yaml)
- rolling foresight for dispatch only (new config now removed)
- browfield with rolling horizon (related to this comment above, new config now removed)
- perfect foresight is out of the list https://github.com/PyPSA/pypsa-eur/blob/efb69ee0be3c4594a6d2e305333d50ede21604ad/scripts/prepare_perfect_foresight.py#L38-L40