pypsa-usa
pypsa-usa copied to clipboard
PyPSA
Edge case clustering issues
Version Checks (indicate both or one)
-
[x] This bug exists on the master branch of PyPSA-USA.
-
[x] This bug exists on the develop branch of PyPSA-USA.
The Issue
I am getting a clustering error on the simplification step when running Wisconsin and Ohio independently at reeds_zone resolution. This is quite similar to the issue discussed in Issue #646.
Steps To Reproduce
Run this config:
# docs :
scenario:
interconnect: [eastern] #"usa|texas|western|eastern"
clusters: [7a] # or 7, 7m, 6, 6a, 6m
simpl: [70]
opts: [3h]
ll: [v1.0]
sector: "" # G
planning_horizons: [2030]
foresight: 'perfect' # myopic, perfect
model_topology:
transmission_network: 'reeds' # [reeds, tamu]
topological_boundaries: 'reeds_zone' # [county, reeds_zone, state]
interface_transmission_limits: false
include: # mixed zone types not supported
# reeds_zone: []
reeds_state: ['WI'] # or "OH"
# reeds_ba: []
aggregate: # eligible keys: [reeds_zone, trans_reg]
# trans_grp: []
# reeds_zone: []
Error Message
Traceback (most recent call last):
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
return self._engine.get_loc(casted_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
raise KeyError(key) from err
KeyError: '1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3053, in get_loc
return self._engine.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1132, in __getitem__
result = self._get_value(key)
^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1237, in _get_value
loc = self.index.get_loc(label)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3055, in get_loc
raise KeyError(key) from err
KeyError: ('p75', '1')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
return self._engine.get_loc(casted_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
raise KeyError(key) from err
KeyError: '1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3220, in _get_loc_level
return (self._engine.get_loc(key), None)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmpbb5f37nj.simplify_network.py", line 292, in <module>
clustering = clustering_for_n_clusters(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 334, in clustering_for_n_clusters
busmap = busmap_for_n_clusters(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 314, in busmap_for_n_clusters
.apply(busmap_for_country, include_groups=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1819, in apply
return self._python_apply_general(f, self._obj_with_exclusions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1885, in _python_apply_general
values, mutated = self._grouper.apply_groupwise(f, data, self.axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 919, in apply_groupwise
res = f(group)
^^^^^^^^
File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 290, in busmap_for_country
n_clusters_per_region[x.name],
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1142, in __getitem__
return self._get_values_tuple(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1210, in _get_values_tuple
indexer, new_index = self.index.get_loc_level(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3150, in get_loc_level
loc, mi = self._get_loc_level(key, level=level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3222, in _get_loc_level
raise KeyError(key) from err
KeyError: ('p75', '1')
Anything else?
The best I can tell is it is an issue with the subnetworks be totally empty. For example, in WI, the subnetwork ("p75", "1") does not exist in the data structure.
When I inspect it, the sub_network, it consists of two buses (25682 and 20889 in this case)
However, these buses are only connected to each other, and have generators, but no loads attached to them.
My best guess is since this is an isolated sub_network with no load, something odd is happening in the busmap clustering? But I don't know the clustering details well enough off the top of my head to make a guess more insightful then that haha.
Pretty hacky solution, but to at least get it to run you can just delete the subnetwork from the network, and then re-determine the network topology. This is a band-aid solution tho, and doesnt really address why this error is happening.
for sub_network in n.buses.sub_network.unique():
buses = n.buses[n.buses.sub_network == sub_network]
loads = n.loads[n.loads.bus.isin(buses.index)]
if loads.empty:
logger.warning(f"Removing sub_network {sub_network}")
gens = n.generators[n.generators.bus.isin(buses.index)]
lines = n.lines[n.lines.bus0.isin(buses.index) | n.lines.bus1.isin(buses.index)]
n.mremove("Bus", buses.index)
n.mremove("Generator", gens.index)
n.mremove("Line", lines.index)
n.determine_network_topology()
As @ktehranchi suggested, changing the weighting-strategy in the clustering.simplify_network config may resolve this issue. I was running with demand-capacity but did not try population.
https://github.com/PyPSA/pypsa-usa/blob/105edcb08df3a9906b5c9459befd2ce44460e7b8/workflow/repo_data/config/config.default.yaml#L158-L163