Improve inputs for buildings retrofit calculations

[ekatef]

Changes proposed in this Pull Request

The PR is intended to update building stock data used in calculations for retrofitting costs. The following changes are suggested.

1. Integrate updated hotmaps data taken from gitlab repo.
2. Removes a patch for Poland, which may be replaced by a recent fix of data structure: the problem for Poland was caused by using a singular form for Construction features (U-value) instead of plural.

Checklist

• [x] I tested my contribution locally and it seems to work fine.
• [ ] Code and workflow changes are sufficiently documented.
• [ ] Changed dependencies are added to envs/environment.yaml.
• [ ] Changes in configuration options are added in all of config.default.yaml.
• [ ] Changes in configuration options are also documented in doc/configtables/*.csv.
• [x] A release note doc/release_notes.rst is added.

[ekatef]

Not sure about a proper strategy to add the updated hotmaps dataset to the repo.

Data inputs for retrofitting are committed into data/retro/ and we could add the updated hotmaps there, but currently data folder is git-ignored. Which probably implies that this way is not a preferable one.

Apart of that, not sure it we want to keep the previous version of hotmaps dataset, or replace it with an updated version.

@martacki what is you vision of the points above?

[ekatef]

The PR was put on hold for quite a while due to some anomalies found in the response on the data update. After digging into the data, it has been found that the anomalies are linked with the structure of the data updates itself.

The PR is ready for review, but may need some additional local testing to ensure that it's fully consistent with all the latest changes.

Update: the local testing successful with an additional pandas-related fix.

@martacki please, feel free to review, when you have time 🙂

[ekatef]

Update: added a number of additional maintenance fixes to address recent pandas deprecations:

• re-implemented a previous fix for axis=1 deprecation in groupby( ) to avoid breaking the data workflow;
• fix a chained assignment warning triggered by inplace in rename( );
• fix some stack deprecation warnings by setting future_stack=True, except one case when we are using dropna=False. I don't quite get how should stack behaviour change in the future version, but it would for sure effect treatment of NaNs, so it feels risky to give a consent for these changes.

[ekatef]

As a comment, these updates of hotmaps data are in fact application of some country-specific scaling coefficients. That is an example of how data structure looks like:

The testing has confirmed that these changes are accurately transferred to the building stock data in the model, and effect mainly the overall area data floor_area_elec_s_{n_clusters}.csv.

[martacki]

Had also a look into the capital_costs of the prepared network (not the EUR/m2, but the resulting costs after multiplying the costs per m2 with the floor area), and here's the result - which will have the biggest impact (annualised costs) on the results: (positive means the costs are now lower, negative numbers mean they're higher).

looks in line with the plot you posted, however I'm a bit worried about the tremendous increase of costs with the new update, retrofitting will be 60-100% more expensive in approx. half of all the countries...

[fneum]

Let me know when this should be reviewed to go into the master. It would be good to have a rough sense for where the cost differences come from.

The added input file should definitely not be part of the git repository.

I would switch to a remote input and also purge the file from the git history:

Purging

pip install git-filter-repo
git filter-repo --path data/retro/data_building_stock.csv --invert-paths
git push origin --force --all

Remote Input

rule ...:
    input:
        building_stock=storage(
            "https://gitlab.com/hotmaps/building-stock/-/raw/master/data/building_stock.csv",
            keep_local=True,
        ),