Clustering: build renewable profiles and add all assets after clustering

[fneum]

Closes #1189.

Changes proposed in this Pull Request

• Moved rules simplify_network and cluster_network before add_electricity and build_renewable_profiles
• Split rule build_renewable_profiles into two separate rules, determine_availability_matrix for land eligibility analysis and build_renewable_profiles, which now only computes the profiles and total potentials from the pre-computed availability matrix.
• Removed variables weight and potential from output of build_renewable_profiles as it is no longer needed
• HAC-clustering is now based on wind speeds and irradiation time series rather than capacity factors
• Added new rule build_hac_features that aggregates cutout weather data to base regions
• Removed {simpl} wildcard and all associated code of the m suffix of the {cluster} wildcard. This means that the option to pre-cluster the network in simplify_network was removed. It will be superseded by clustering renewable profiles and potentials within clustered regions by resource classes.
• The rule build_line_rating now directly attaches the DLR profiles to the pypsa.Network which is then returned as output.
• The rule add_extra_components was integrated into add_electricity
• Added new rule build_electricity_demand_base to determine the load distribution of the substations in the base network (which was previously done in add_electricity). This time series is used as weights for kmeans-clustering in cluster_network and is later added to the network in add_electricity in aggregated form.
• The weights of the kmeans clustering algorithm are now exclusively based on the load distribution. Previously, they also included the distribution of thermal capacity.
• Since the networks no longer start with the whole electricity system added pre-clustering, the files have been renamed from elec...nc to base...nc.
• The scripts simplify_network.py and cluster_network.py were simplified to become less nested and profited from the removed need to deal with cost data.
• New configuration options to calculate connection costs of offshore wind plants.

Updated Workflow Graph

Testing

To be done.

A first observation is that build_renewable_profiles on clustered regions is much faster!

Without a doubt, this PR is not going to be backward compatible, which is why it should be pulled after a v0.12.0 release.

Next steps

• Cluster renewable profiles and potentials within clustered regions by resource classes, emulating the previous m option in {clusters} wildcard, but better.
• Add option to cluster by administrative shapes #263.

Checklist

• [x] I tested my contribution locally and it seems to work fine.
• [x] Code and workflow changes are sufficiently documented.
• [x] Changed dependencies are added to envs/environment.yaml.
• [x] Changes in configuration options are added in all of config.default.yaml.
• [x] Changes in configuration options are also documented in doc/configtables/*.csv.
• [ ] A release note doc/release_notes.rst is added.

[cpschau]

Capacity factors (pre-solve)

Loads

Installed capacities

64 clusters

512 clusters

[cpschau]

Shapes

Clusters per country

[FabianHofmann]

@fneum this is awesome. One minor comment about the infixes (will have a deeper look when there is time). I was wondering why we keep the _s_ infix. It seems that this is because the dlr is still done before clustering. However, I doubt that this actually makes much sense. First calculating dlr and clustering, is probably as bad as first clustering and then calculating dlr. The reason being that one does not know which of the clustered transmission lines are parallel and which are in series.

[lisazeyen]

Change in capital_cost for offwind depending on later or earlier clustering for 37 nodes

[lisazeyen]

Changes in p_nom_max. In general, seems to be quite similar higher solar potential in Norway

Warning for the following nodes that installed capacity is larger than technical potential ['BE1 0 offwind-ac-2025', 'DE1 0 offwind-ac-2025', 'GB5 0 offwind-ac-2025', 'NL1 0 offwind-ac-2025', 'ES4 0 onwind-2025', 'ES4 0 solar-2025']

[lisazeyen]

Change in offwind-acprofiles for 37 nodes

[lisazeyen]

Change in offwind-acprofiles for 37 nodes

this is not important anymore since underwater fraction is removed in this PR

[lisazeyen]

Looks good to me now! Just the offwind-float capital cost seem to increase somewhat. Amazing, that the investment costs are so similar now even though the calculation is quite different...

[lisazeyen]

When checking the p_max_pu for onshore and offshore wind I see some larger deviations, averaged over all 38 nodes and all time steps:

For example for Belgium, 38 nodes, wind seems to have always higher capacity factors in the feature branch

Similar in Austria onshore wind

[lisazeyen]

run of river in LU is missing in feature branch, Italy as slightly larger capacities. Overall summed capacity of ror stays constant.

[fneum]

I could not reproduce the divergence in capacity factor time series. All looks good from my side; I checked:

• average capacity factors per carrier and country
• average loads per country
• existing powerplants capacity per country
• capital_cost per carrier and country
• installable potential per carrier and country
• total network volume by DC and AC

All values are acceptably similar.

Indeed closes #1216.

[fneum]

Ready to be merged from my side @lisazeyen, @cpschau, @FabianHofmann. Most tests pass, I extensively tested and compared the outputs. Now is a good time as there are not too many other open PRs and there was a recent release. It's much faster :)

[lisazeyen]

Ready to be merged from my side @lisazeyen, @cpschau, @FabianHofmann. Most tests pass, I extensively tested and compared the outputs. Now is a good time as there are not too many other open PRs and there was a recent release. It's much faster :)

Great, looks good to me!

[cpschau]

Ready to be merged from my side @lisazeyen, @cpschau, @FabianHofmann. Most tests pass, I extensively tested and compared the outputs. Now is a good time as there are not too many other open PRs and there was a recent release. It's much faster :)

Great! Looking very much forward to the fast RES profiles build!

[FabianHofmann]

let's do it! :rocket: