Create a simple methodology to create the demand time series for countries with empty values

[davide-f]

In the build_demand script, adding an option and code that creates a time series for missing countries or countries with no demand (0.0).

[davide-f]

@ekatef @SermishaNarayana @mdzzg

[ekatef]

As a little practical hint, a nice dataset on electricity demand is provided by Ember: https://ember-climate.org/data/data-tools/data-explorer/. All Ember datasets are released under the creative common license 🙏🏽 Which means that we can modify and re-distribute them as we feel like, provided a proper acknowledgement is provided.

As the first idea for restoring missed time-series, we can probably use demand time-series of the neighbouring countries as scale them using Ember data. @SermishaNarayana What do you think?

[SermishaNarayana]

@ekatef so instead of using GDP and population data of the country where data is missing, we use the Ember data to attain the scaling factor. Then apply this scaling factor to the demand time series of the neighbouring countries? Did I understand it correctly?

[ekatef]

@ekatef so instead of using GDP and population data of the country where data is missing, we use the Ember data to attain the scaling factor. Then apply this scaling factor to the demand time series of the neighbouring countries? Did I understand it correctly?

Yeah, absolutely! I think, we can directly use the annual demands from Ember for countries where we currently miss the energy demand. If that is not the case, and the country does not present in Ember data (e.g. Western Sahara) we may need to use GDP and population to restore the annual demand. But it would require more efforts, so I'd leave that as a next step.

Does it make sense for you?

[SermishaNarayana]

@ekatef Yes, perfectly

[ekatef]

@ekatef Yes, perfectly

Super, then we seem to be well aligned :) As the next step, I'd suggest to identify for which countries we can fill GEGIS gaps using Ember data, and for which years are data available for them and countries "analogues" which we can use to restore the missed time-series.

It can be also worth to think which correction coefficients can be applied to the available time-series to account for the differences in the demand dynamics for the given country (with missed data) and a country-analogue wee are using to restore demand. The most obvious factors which may need correction:

• a share of the constant demand (I'd expect that industry demand will be more or less constant in time);
• difference in weather (that is important as the electricity demand is known to have a pronounced weather signature).

What do you think @SermishaNarayana @davide-f ?

[Tomkourou]

Working on this downstream and have already identified the countries where this is an issue. Around 48 in total. Happy to support on a fix.

Perhaps this dataset could serve as an alternative for those countries with gaps. Also under a CC0 license -> https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/CBYXBY&version=6.2&selectTab=termsTab

ISO Code	Full Country Name
AF	Afghanistan
AG	Antigua and Barbuda
AS	American Samoa
AW	Aruba
BB	Barbados
BI	Burundi
BS	Bahamas
BT	Bhutan
BZ	Belize
CK	Cook Islands
CV	Cape Verde
DM	Dominica
FK	Falkland Islands
FO	Faroe Islands
GD	Grenada
GF	French Guiana
GI	Gibraltar
GP	Guadeloupe
GU	Guam
GY	Guyana
KM	Comoros
KN	Saint Kitts and Nevis
KY	Cayman Islands
LA	Laos
LC	Saint Lucia
MQ	Martinique
MS	Montserrat
MT	Malta
MV	Maldives
NC	New Caledonia
NR	Nauru
NU	Niue
PF	French Polynesia
PG	Papua New Guinea
RE	Réunion
SB	Solomon Islands
SC	Seychelles
ST	Sao Tome and Principe
TC	Turks and Caicos Islands
TL	Timor-Leste (East Timor)
TO	Tonga
UG	Uganda
VC	Saint Vincent and the Grenadines
VG	British Virgin Islands
VI	U.S. Virgin Islands
VU	Vanuatu
WS	Samoa
XK	Kosovo

[ekatef]

Hello @Tomkourou! Amazing and thanks a lot for offering help.

Your suggestion on using PLEXOS database sounds really great and a contribution on that would be much appreciated. Are you also planing to account for the differences between the weather years? My feeling is that is can be done in quite a straight-forward way.

It would be an amazing and relatively "easy" fix, while @SermishaNarayana could continue to work on a more advanced approach which implies running the full-scale model, as discussed in the first point of #933.

[SermishaNarayana]

Update on GEGIS execution

Most of the datasets to be downloaded in the download_datasets method were hosted on university links and are now inactive.

Most of the inactive links have alternate sources to download them except for the following:

1. Average monthly wind speeds 1979-2019 (Required file - era5monthlywind.h5)
2. Average monthly solar insolation 1979-2019 (Required file - era5monthlysolar.h5)
3. "Various smaller datasets" - There is not much description I found of what entails the smaller datasets but since the link is inactive (I am not sure what these datasets are). Probably further execution of GEGIS may help in finding these requirements

[Tomkourou]

FYI we implemented plexos load profiles in our fork here -> https://github.com/transition-zero/pypsa-earth-trace/blob/main/scripts/build_demand_profiles.py

It's a bit of a quick and dirty fix due to time pressures but happy to work with someone to integrate it more thoughtfully into pypsa-earth if that is of interest.

[ekatef]

FYI we implemented plexos load profiles in our fork here -> https://github.com/transition-zero/pypsa-earth-trace/blob/main/scripts/build_demand_profiles.py

It's a bit of a quick and dirty fix due to time pressures but happy to work with someone to integrate it more thoughtfully into pypsa-earth if that is of interest.

Cool! 😄 Thanks for letting know @Tomkourou

The approach with plexos data looks reasonable. Great work! I wonder if plexos allows re-distributions of the load dataset. Is the dataset open-licensed, correct?

[mdzzg]

@ekatef EE datasets are a product of theirs: https://www.energyexemplar.com/datasets

Which they sell to the clients (read: companies) already using a PLEXOS license:

@ekatef has brough up a very valid point (anything coming from Katia is!).

[Tomkourou]

I don't think the data is directly from plexos. It was a dataset used in this paper where plexos was used to run the model. The dataset in question can be found here under a CC0 license + citation requirements. Apologies I realised I just called it a plexos dataset which is confusing 😅

[mdzzg]

I used this dataset by Maarten Brinkerink. It is high quality/superb, and Maarten is a stand-up guy, technically speaking.

Well done Thomas.